search

mberkay0 / pretrained-backbones-unet

A PyTorch-based Python library with UNet architecture and multiple backbones for Image Semantic Segmentation.
MIT License
46 stars 9 forks source link

Training loss is not decreasing #5

Open SamratSinghRathore opened 1 year ago

SamratSinghRathore commented 1 year ago

Hello, I am using Unet decoder and HRNet encoder model but i realize that the training loss is not decreasing, instead it seems to remain the same. Also the training logs appear in separate lines which is a problem becuase 1 epoch take around 200 lines and not gets output in one line. Screenshot from 2023-06-16 19-27-31

I have trained it for 10 epochs but the loss keeps jumping between 0.99 and 1

mberkay0 commented 1 year ago

Greetings @SamratSinghRathore, The divergence/non-convergence of a model may depend on more than one situation. I recommend trying different parameters and different encoders. You may also consider freezing specific layers of the encoder. Another option is to freeze the entire encoder and train only the decoder. In addition, please ensure to pre-process your data according to the backbone network model you have chosen. Best regards.

Note: The separate line verbose is due to the 'tqdm' library. This problem may occur in the Jupyter Notebook. I will update the 'tqdm.auto' module. Perhaps you can use this for a quick solution:


from tqdm import tqdm

tqdm._instances.clear()
SamratSinghRathore commented 1 year ago

Hello,

Thank you for the reply. Will implement and get back to you.

Regards Samrat

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Berkay Mayalı @.> Sent: Saturday, June 24, 2023 6:24:21 PM To: mberkay0/pretrained-backbones-unet @.> Cc: SamratSinghRathore @.>; Mention @.> Subject: Re: [mberkay0/pretrained-backbones-unet] Training loss is not decreasing (Issue #5)

Greetings @SamratSinghRathorehttps://github.com/SamratSinghRathore, The divergence/non-convergence of a model may depend on more than one situation. I recommend trying different parameters and different encoders. You may also consider freezing specific layers of the encoder. Another option is to freeze the entire encoder and train only the decoder. In addition, please ensure to pre-process your data according to the backbone network model you have chosen. Best regards.

Note: The separate line verbose is due to the 'tqdm' library. This problem may occur in the Jupyter Notebook. I will update the 'tqdm.auto' module. Perhaps you can use this for a quick solution:

from tqdm import tqdm

tqdm._instances.clear()

— Reply to this email directly, view it on GitHubhttps://github.com/mberkay0/pretrained-backbones-unet/issues/5#issuecomment-1605463306, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXX3EZGHLAZRX5XZ4SK3TXLXM3PP3ANCNFSM6AAAAAAZJJOIMM. You are receiving this because you were mentioned.Message ID: @.***>