search

Westlake-AI / openmixup

CAIRI Supervised, Semi- and Self-Supervised Visual Representation Learning Toolbox and Benchmark
https://openmixup.readthedocs.io
Apache License 2.0
624 stars 62 forks source link

Pre-trained model of A2MIM #13

Closed zugofn closed 1 year ago

zugofn commented 2 years ago

Hi, when will the pre-trained model of A2MIM be released in the webpage? The work is so meaningful！ Thank you!

Lupin1998 commented 2 years ago

Hi, @zugofn. Thanks for your appreciation of A2MIM. Since A2MIM is under review, we plan to release the pre-trained model in late September or later. We will inform you when we update the release. Thanks again for your attention and patience!

zugofn commented 2 years ago

Hi, @zugofn. Thanks for your appreciation of A2MIM. Since A2MIM is under review, we plan to release the pre-trained model in late September or later. We will inform you when we update the release. Thanks again for your attention and patience!

Thanks for your quick reply！ Have a nice day😊

donggoing commented 2 years ago

Hi, @zugofn. Thanks for your appreciation of A2MIM. Since A2MIM is under review, we plan to release the pre-trained model in late September or later. We will inform you when we update the release. Thanks again for your attention and patience!

Is it ok to release the model parameters now? I would like to do research based on this paper.

Lupin1998 commented 2 years ago

Hi, @zugofn. Thanks for your appreciation of A2MIM. Since A2MIM is under review, we plan to release the pre-trained model in late September or later. We will inform you when we update the release. Thanks again for your attention and patience!

Is it ok to release the model parameters now? I would like to do research based on this paper.

Hi, @donggoing. Thanks for your attention. Unfortunately, the pre-trained models cannot be released now because we are working on the revision and resubmission of A2MIM. We are sorry about the late release and we plan to upload the pre-trained models after the deadline for ICLR2023 submission.

Meanwhile, feel free to contact me if you have some questions about A2MIM. We can also discuss the relevant questions encountered in your research.

donggoing commented 2 years ago

Hi, @zugofn. Thanks for your appreciation of A2MIM. Since A2MIM is under review, we plan to release the pre-trained model in late September or later. We will inform you when we update the release. Thanks again for your attention and patience!

Is it ok to release the model parameters now? I would like to do research based on this paper.

Hi, @donggoing. Thanks for your attention. Unfortunately, the pre-trained models cannot be released now because we are working on the revision and resubmission of A2MIM. We are sorry about the late release and we plan to upload the pre-trained models after the deadline for ICLR2023 submission.

Meanwhile, feel free to contact me if you have some questions about A2MIM. We can also discuss the relevant questions encountered in your research.

Thanks for your quick reply! ”after the deadline for ICLR2023 submission“ you mean like after Sep 28? And i would like to know how do you think about what brings the gap between Tr（A2MIM） and CNN（A2MIM）?

Lupin1998 commented 2 years ago

Yes, @donggoing, it might be uploaded in early October. As for your question, A2MIM works for Transformer and CNN, and there is no difference in its algorithm for both two architectures. When Transformer and CNN adopt A2MIM using the same pre-training epochs, the performance gap might come from two aspects:

(1) Transformer architectures are more suitable for MIM pre-training than CNNs. I have pre-trained A2MIM for longer epochs based on both architectures, ViT-B obtains slight performance gains (800 epochs vs. 1200 epochs), while ResNet-50 suffers from performance degeneration (300 epochs vs. 600 epochs). From my perspective, CNNs have more substantial inductive bias than Transformers but have fewer degrees of freedom. MIM empowerer Transformers with the relevant inductive bias, which helps Transformers to achieve better fine-tuning performances.

(2) The design gap of architecture between ResNet-50 and ViT might cause the sub-optimal performances of CNNs. As you know, ViT belongs to MetaFormer architectures, which is more powerful than ResNet (the vanilla bottleneck module). In A2MIM, we only verify ResNet-50 and ViT architectures. If you adopt advanced ConvNet architectures (e.g., ConvNeXt), you might find Transformers and CNNs yield similar pre-training performances.

donggoing commented 2 years ago

@Lupin1998 I see. Thanks for your detailed reply! Hopefully, the model parameters can be released as soon as possible.

ksnzh commented 1 year ago

Waiting for pre-trained weights.

Lupin1998 commented 1 year ago

Hi guys, thanks for your attention and patience. We have uploaded the release of weights, logs, visualizations of A2MIM and relevant self-supervised methods in a2mim-in1k-weights. We are sorry for the delay and hope you will find it useful. Feel free to ask me if you have further questions.

ksnzh commented 1 year ago

@Lupin1998 Thanks for your work. I tried to find the self-supervised pre-training weight of a2mim without finetune. Does full_a2mim_r50_l3_sz224_init_8xb256_cos_ep100 stand for that?

Lupin1998 commented 1 year ago

@Lupin1998 Thanks for your work. I tried to find the self-supervised pre-training weight of a2mim without finetune. Does full_a2mim_r50_l3_sz224_init_8xb256_cos_ep100 stand for that?

Yes, weights starting with full contain the full weights of pre-training models, while those with backbone are the backbone-only weights (can be used for downstream tasks in PyTorch implementations).

Lupin1998 commented 1 year ago

Hi, guys @zugofn, @donggoing, @ksnzh. Since we have uploaded the release a2mim-in1k-weights and updated documents in issue #31, we will close this issue in two days if there are no new questions. You can reopen it or start a new issue if you have more questions. I am pleased to discuss research topics relevant to self-supervised learning with you! 😃

donggoing commented 1 year ago

@Lupin1998 Ok~Thanks for your work！

zugofn commented 1 year ago

Hi, guys @zugofn, @donggoing, @ksnzh. Since we have uploaded the release a2mim-in1k-weights and updated documents in issue #31, we will close this issue in two days if there are no new questions. You can reopen it or start a new issue if you have more questions. I am pleased to discuss research topics relevant to self-supervised learning with you! 😃

Thanks for your quick and patient reply, thank you for your work! Happy weekend~😀

Lupin1998 commented 1 year ago

I closed this issue if there is no more question. You can reopen it or start a new issue if you have more questions. Thanks again for your attention.