Closed silent357 closed 3 years ago
Did you reinstall mmcv after you've updated PyTorch? Also, rerunning the training process might help as sometimes bad initialization may occur.
It seems that FCENet needs more tests
Did you reinstall mmcv after you've updated PyTorch? Also, rerunning the training process might help as sometimes bad initialization may occur.
Yes, I reinstalled mmcv after updating PyTorch. The NaN losses occur every time I run the FCENet training on ICDAR2015.
It seems that FCENet needs more tests
Maybe.
Are you using mmdet==2.13.0
? We only support mmdet==2.11.0
at this time but will fix it soon.
I change the version of mmdet to 2.11.0 and it works! Thank you!
I have another question. I find that when I train an fcenet model with 2gpus, I cannot train other models with multiple gpus. Sometimes I cannot either test an fcenet model. However, it is okay to train a model with 1 gpu when I train an fcenet model with 2 gpus. The error is as below:
You can modify the port number at https://github.com/open-mmlab/mmocr/blob/main/tools/dist_train.sh#L13
btw please post a new issue for new questions
@gaotongxiao if you have interest, the default port number can be made random. Something like $((12000 + $RANDOM % 20000))
OK. Thank you!
When I install mmdet==2.11.0, it occur an error: ModuleNotFoundError: No module named 'mmdet.datasets.api_wrappers'
@liangxiaoyun mmdet 2.11.0 has not been officially supported since 0.2.1. Please make sure your mmdet version fits the requirement here
Hi, I train FCENet on ICDAR2015 but get NaN for all the losses. The training seems proper on CTW1500. Is there anything wrong?