When I change the code , the result will dropp considerably！

[Zhangwenyao1]

Thanks for your great work！ The previous issue has been solved， but I find a new issue. If I change the code (https://github.com/muzairkhattak/multimodal-prompt-learning/blob/89fecdc38a38712ea740df25aca3826b63539194/trainers/coop.py#L251) as self.register_model("model", self.model, self.optim, self.sched). The result will drop considerably(20%)！ Can you give me some advice?

[muzairkhattak]

Hi @Zhangwenyao1,

Thank you for showing interest in our work.

The code line that you have mentioned just registers the model, optimizer and the scheduler in Dassl trainer class. https://github.com/KaiyangZhou/Dassl.pytorch/blob/c61a1b570ac6333bd50fb5ae06aea59002fb20bb/dassl/engine/trainer.py#L86

Later on, it re-uses those instances for loading (for inference) and saving (after training) of weights. https://github.com/KaiyangZhou/Dassl.pytorch/blob/c61a1b570ac6333bd50fb5ae06aea59002fb20bb/dassl/utils/torchtools.py#L77

You have mentioned that

The result will drop considerably(20%)！

Can you please clarify the following queries?

• Under which setting you are training CoOp's model and which config file are you using? Is it few-shot training or base-to-novel generalization setting?
• If you are using base-to-novel setting, then the results drops for base classes or for the novel classes?
• Can you please clarify if this result drops when you load the saved model and then test it, or it happens during the evaluation that occurs after training (without loading the saved model from directory)?

Kind regards.

[Zhangwenyao1]

Thanks for your reply!

1. Under which setting you are training CoOp's model and which config file are you using? Is it few-shot training or base-to-novel generalization setting?

Except register_model(self.model.promptlearner,,_)->registermodel(self.model,,_), I did not change other code and config files(rn50.yaml). It is 16-shot training setting （dataset: Oxford_flowers）.

2.If you are using base-to-novel setting, then the results drops for base classes or for the novel classes?

I used base setting rather than base-to-novel setting

3. Can you please clarify if this result drops when you load the saved model and then test it, or it happens during the evaluation that occurs after training (without loading the saved model from directory)?

The accuracy of both training and evaluation drops as shown below:

*epoch [198/200] batch [25/25] time 0.205 (0.240) data 0.000 (0.038) loss 0.9525 (0.9364) acc 71.8750 (76.4375) lr 1.1101e-06 eta 0:00:12 Evaluate on the val* set => result

• total: 408
• correct: 287
• accuracy: 70.3%
• error: 29.7%
• macro_f1: 69.4%**

*epoch [200/200] batch [25/25] time 0.200 (0.242) data 0.000 (0.042) loss 0.8314 (0.9537) acc 76.5625 (74.5625) lr 1.2337e-07 eta 0:00:00 Evaluate on the val* set => result

• total: 408
• correct: 295
• accuracy: 72.3%
• error: 27.7%
• macro_f1: 71.6%**

Thanks again for your help.

[muzairkhattak]

Hi @Zhangwenyao1,

I'm sorry for the late reply.

I have checked the coop.py file.

https://github.com/muzairkhattak/multimodal-prompt-learning/blob/89fecdc38a38712ea740df25aca3826b63539194/trainers/coop.py#L251

I am also encountering the same issue as yours. After debugging, I found that this effect is coming from the dassl library.

Dassl library turns on the training mode for the model part (self._models[name].train() at this line) that is explicitly registered.

To resolve the issue, you can comment out this line.

Eventually, if you are planning to train any other part of CLIP model then you will need to put the model part in to the train mode.

I hope this will resolve your current issue.

Kindly let me know if you have any further queries.

[Zhangwenyao1]

Thanks a million. This issue has bothered me for a long time, thanks again for your help, and look forward to your follow-up work.

[muzairkhattak]

I'm glad that the issue has been resolved.

You are always welcome!

Thank you.