search

datvuthanh / HybridNets

HybridNets: End-to-End Perception Network
MIT License
582 stars 118 forks source link

Problem in Training stage #59

Closed dreamer-1996 closed 2 years ago

dreamer-1996 commented 2 years ago

Hello, I tried to follow your suggestion to train the model. So accordingly, at first I freeze the segmentation and trained for some epoch.

python train.py -p bdd100k -c 3 -n 4 -b 8 --freeze_seg True --lr 1e-5 --optim adamw --num_epochs 75 --val_interval 1 --log_path D:\HybridNets\rgb-clean --saved_path D:\HybridNets\rgb-clean --save_interval 500 --verbose True --num_gpus 1 --plots True

After that I am freezing the backbone and detection head.

python train.py -p bdd100k -c 3 -n 4 -b 8 --freeze_backbone True --freeze_det True --lr 1e-5 --optim adamw --num_epochs 12 --val_interval 1 --log_path D:\HybridNets\rgb_clean --saved_path D:\HybridNets\rgb_clean --save_interval 500 --verbose True --num_gpus 1 --plots True -w D:\HybridNets\rgb-clean\bdd100k\hybridnets-d3_74_129225_best.pth

But I am getting the error

image

Can you please suggest how to solve this issue?

Thank you in advance

xoiga123 commented 2 years ago

Sorry for the late reply, there was a bug in val.py, I'll fix in next commit.

In the mean time, please change this line https://github.com/datvuthanh/HybridNets/blob/c15ba5b3c019c8a7fea71e017c18c05b163adb18/train.py#L203 to optimizer.load_state_dict(ckpt['optimizer'].state_dict()) and it'll load the optimizer from weight correctly.

dreamer-1996 commented 2 years ago

Thank you for your reply.