Closed oopsboy closed 2 years ago
I have the same problem as you, did you solve it? and my total loss became nan
@wha-cloud Could you check if the learning rate is too large? Or if you use too small batch size? See: https://github.com/facebookresearch/adaptive_teacher/issues/9
@wha-cloud Could you check if the learning rate is too large? Or if you use too small batch size? See: #9
Thanks! When I set the batch size to 8, up to now 2000 iterations are no problem. If it is set to 4, there are problems with several hundred iterations. Now I have a new question, it seems to be evaluated twice when training, why is the result of the second evaluation 0?
@wha-cloud I wonder if you have solved it?
@wha-cloud @oopsboy Does the number in tensorboard make sense? I always looked at the number in tensorboard and rarely see the numbers printed out.
@wha云@oopsboy张量板中的数字有意义吗?我总是在张量板中查看数字,很少看到打印出来的数字。
@yujheli Yes, when I test with the trained model, its map is also 0, so this problem exists
@wha-cloud I guess you are in the source only pre-training stage, in which the student model does not have learned weights yet (https://github.com/facebookresearch/adaptive_teacher/blob/cba3c59cadfc9f1a3a676a82bf63d76579ab552b/adapteacher/engine/trainer.py#L506).
@wha-cloud I guess you are in the source only pre-training stage, in which the student model does not have learned weights yet (
).
@helq2612 Thank you for your answer. How can I modify the number of times of this paramete?
@oopsboy I think you can modify it in the config files, e.g.: https://github.com/facebookresearch/adaptive_teacher/blob/cba3c59cadfc9f1a3a676a82bf63d76579ab552b/configs/faster_rcnn_R101_cross_clipart.yaml#L43 Or pass it as args to train_net.py
Ok, the model will be evaluated two times: If you are in the stage of burn-in, you will get 0 AP for teacher. bbox (not uses):
bbox_student:
ok,3q!
Train