Introducing mmselfsup.core in mmdet leads to unexpected results.

[liming-ai]

Hi @ZwwWayne @BIGWangYuDong @RangiLyu @jbwang1997 @hhaAndroid @chhluo , happy mid-Autumn Festival (🐶)~

This issue also open in mmselfsup

Thanks for reporting the unexpected results and we appreciate it a lot.

Checklist

1. I have searched related issues but cannot get the expected help.
2. The unexpected results still exist in the latest version.

Describe the Issue I tried to combine mmselfsup and mmdet to do something, however, when I added:

custom_imports = dict(imports=['mmselfsup.core'])

I found that DETR's training fails to converge, I also added this line into other methods like fcos/retinanet/mask-rcnn. In the case of fixed random seeds, their loss values and training results have huge changes. I'm clueless about this and what's causing the huge discrepancy, hoping to get some help from the community.

Reproduction

1. What command, code, or script did you run?

Add this line into any original mmdet config, you can find the loss and result changed a lot, even with fixed random seed.

custom_imports = dict(imports=['mmselfsup.core'])

2. Did you make any modifications on the code? Did you understand what you have modified? Yes. Yes.

Bug fix If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

[jbwang1997]

Hello @mitming Could you provide the version of mmdetection and mmselfsup you used? I would like to reproduce this bug and explore the problem.

[liming-ai]

Hello @mitming Could you provide the version of mmdetection and mmselfsup you used? I would like to reproduce this bug and explore the problem.

@jbwang1997 Thanks a lot! Please try DETR and Deformable DETR first, they are failed to converge with nan or inf grad_norm.

Package         Version    Source
--------------  ---------  ----------------------------------------------
mmcls           0.23.2     https://github.com/open-mmlab/mmclassification
mmcv-full       1.6.1      https://github.com/open-mmlab/mmcv
mmdet           2.25.1     https://github.com/open-mmlab/mmdetection
mmsegmentation  0.27.0     http://github.com/open-mmlab/mmsegmentation
mmselfsup       0.9.2      https://github.com/open-mmlab/mmselfsup

[jbwang1997]

I think the problem is because mmselfsup force register their DefaultOptimizerConstructor into OPTIMIZER_BUILDERS, which covers the original DefaultOptimizerConstructor in mmcv.

Both DETR and Deformable DETR need to set custom_keys, which only make effect in mmcv DefaultOptimizerConstructor.

[jbwang1997]

This bug needs to be fixed in mmselfsup, It's better also to report it there.

BTW, this bug will not occurs if you use the newest mmdet 3.0 and mmselfsup 1.0

[liming-ai]

@jbwang1997 Thanks for your answer, in my experiment, this bug also affects other methods like mask-rcnn/fcos/retinanet, could you please check again?

[liming-ai]

This bug needs to be fixed in mmselfsup, It's better also to report it there.

BTW, this bug will not occurs if you use the newest mmdet 3.0 and mmselfsup 1.0

@jbwang1997, I have no idea why this bug affect other methods like Mask-RCNN and RetinaNet, since they do not need use custom_keys, could you please check that?

[jbwang1997]

Hi @mitming. I'm trying to figure out the problem. It may take some time. If I find out the problem, I will report the results in this issue as soon as possible.

[liming-ai]

Hi @mitming. I'm trying to figure out the problem. It may take some time. If I find out the problem, I will report the results in this issue as soon as possible.

Thanks a lot! Looking forward to your reply!