Closed letme-hj closed 1 year ago
Hi~Thanks for your interest in our work. It seems that the network doesn't learn anything (the loss curve doesn't make any changes). I reconfirmed that the version of MMsegmentation is 0.27.0
. Please try run for scratch to confirm the correctenss of the whole workflow (lie the path of data, model, etc.). And then make sure that the pre-trained checkpoint is loaded correctly. If this issue still exits, I hope you can share the log and configuration files. I would try to see if there are any problems with them.
Thanks again :)
Hi, thank you for the quick response!
Actually, I already tried to train from scratch and it still does not work. Here is the config file and log file:
I believe the pretrained weights are loaded properly, because I get this message
backbone.layer2.2.bn2.weight - torch.Size([128]):
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth
backbone.layer2.2.bn2.bias - torch.Size([128]):
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth
backbone.layer2.2.conv3.weight - torch.Size([512, 128, 1, 1]):
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth
backbone.layer2.2.bn3.weight - torch.Size([512]):
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth
backbone.layer2.2.bn3.bias - torch.Size([512]):
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth
...
for all backbone layers when I set checkpoint_path={path of pretrained backbone}
,
while I get this
backbone.layer2.2.bn2.weight - torch.Size([128]):
The value is the same before and after calling `init_weights` of EncoderDecoder
backbone.layer2.2.bn2.bias - torch.Size([128]):
The value is the same before and after calling `init_weights` of EncoderDecoder
backbone.layer2.2.conv3.weight - torch.Size([512, 128, 1, 1]):
KaimingInit: a=0, mode=fan_out, nonlinearity=relu, distribution =normal, bias=0
backbone.layer2.2.bn3.weight - torch.Size([512]):
ConstantInit: val=0, bias=0
backbone.layer2.2.bn3.bias - torch.Size([512]):
The value is the same before and after calling `init_weights` of EncoderDecoder
...
when I train from scratch(checkpoint_path=None
).
Please let me know if there seems to be something wrong in my config. Thank you!
Hi~.
It seems that this error is from computation precision.
I used the mixed precision during my whole experiment (e.g FP16OptimierHook
). However, I mistakenly forgot to change the corresponding code.
I used your same configuration and meet the same problem. All of these process fail to converge whatever training from scratch, from CMID-Backbone, or from ImageNet backbone.
Finally, I have a look at my original configuration, and find the devil:
Change the line:
optimizer_config = None
to
optimizer_config = dict(type='Fp16OptimizerHook', distributed=False)
You can have a try to see if this will be helpful. It's really strange that this error occurs, maybe it's due to a problem in the optimization process. I look forward to your feedback.
Thank you so much!!
I tried and it seems to be working now! Thank you again for your kind response :) 👍
Thanks for your feedback. I will update the corresponding code.
Closed as long periods of inactivity, feel free to reopen if there is any problem.
Hi, Thank you for the work and sharing code! I find it very helpful :)
I have one question though, I am trying to reproduce your result on ResNet50 + Potsdam dataset (Semantic segmentation), so I followed the instructions you mentioned in SemanticSegmentation. (I used your pretrained weights that's trained on MillionAID for the backbone.)
That is,
However I completely fail to train the model...
I wonder if I used the right version of mmsegmentation. (I'm confused since the environment.yml is only in the Pretrain folder) Or is there any other lines that I have to be aware of?
Thank you in advance for your help!