search

yue-zhongqi / ifsl

[NeurIPS 2020] Released code for Interventional Few-Shot Learning
164 stars 23 forks source link

Can't get the reported performance #3

Closed Kyfafyd closed 3 years ago

Kyfafyd commented 3 years ago

Sorry that I can not get the reported performance. I am running the experiment SIB/minires_1_ifsl.yaml, my validation acc is lower than 60%, but reported 68.85%?

Look forward for reply! Thanks!

yue-zhongqi commented 3 years ago

I checked the config file and it looked correct.

  1. Did you download the mini-ImageNet from the link given? (The dataset should contain .png files of size 84*84)
  2. Did you download and use the pre-trained backbone as indicated in the readme.md?
Kyfafyd commented 3 years ago

I checked the config file and it looked correct.

  1. Did you download the mini-ImageNet from the link given? (The dataset should contain .png files of size 84*84)
  2. Did you download and use the pre-trained backbone as indicated in the readme.md?

yes, I just follow the readme instruction.

When I am reproducing the experiments, I only meet a problem: For the following code, this line: https://github.com/yue-zhongqi/ifsl/blob/eea96618dbaa295be569dafb4707686fedc1b5fe/SIB/PretrainedModel.py#L65 It also include fc.weight and fc.bias into themodel_params, in which fc.weight with size [908, 512] mismatch the size of fc.weight ([64, 512]) in model_dict. So, considering this, I just removed fc.weight and fc.bias frommodel_params and load other pretrained params.

I wonder if this is the code wrong? or this is specifically designed?

Thanks for reply and looking forward for further clarification!

yue-zhongqi commented 3 years ago

I didn't get this error with this code base. The last fc layer is the pre-trained classifier. It is used for class-wise adjustment and is quite important. For miniImageNet, the fc layer should map 512-dimension feature to 64 classes. Did you load the correct pre-trained model file because the miniImageNet pre-trained model is for 64 classes not 908 classes?

Kyfafyd commented 3 years ago

I didn't get this error with this code base. The last fc layer is the pre-trained classifier. It is used for class-wise adjustment and is quite important. For miniImageNet, the fc layer should map 512-dimension feature to 64 classes. Did you load the correct pre-trained model file because the miniImageNet pre-trained model is for 64 classes not 908 classes?

Oh, I see. Thanks you a lot. I have double checked that I have loaded the resnet10 pretrained model for miniImageNet, but still get this error...so confusing. So can you please upload the pretrained models you used in your experiments so I can try once more?

yue-zhongqi commented 3 years ago

My server is under maintenance today. Once it is back online, I will help you take a look.

Kyfafyd commented 3 years ago

My server is under maintenance today. Once it is back online, I will help you take a look.

Thanks a lot for keeping attention and sorry for bothering!!!

yue-zhongqi commented 3 years ago

I verified that I didn't get this error. I have uploaded the pre-trained weights. Please try this and let me know if it works. https://drive.google.com/file/d/1wWFQibiLL6yJWNXsHIxioucp0DnxOelP/view?usp=sharing

Kyfafyd commented 3 years ago

I verified that I didn't get this error. I have uploaded the pre-trained weights. Please try this and let me know if it works. https://drive.google.com/file/d/1wWFQibiLL6yJWNXsHIxioucp0DnxOelP/view?usp=sharing

Thanks a lot for uploading the models.

The error mentioned before is solved now. However, I got best acc 66.578 +- 0.569%, can't reach reported 68.85. BTW, I am running SIB/minires_1_ifsl.yaml

I wonder if there are some settings different from experiments in your paper? Hope to see your reply!

yue-zhongqi commented 3 years ago

Hi I have fixed a bug in network.py. I modified the code (in dni_linear) to test SIB in inductive setting but never changed back. Sorry for the inconvenience caused. I tested on the saved model and below is the test accuracy (I didn't fix random seed so higher than paper). image

Kyfafyd commented 3 years ago

Hi I have fixed a bug in network.py. I modified the code (in dni_linear) to test SIB in inductive setting but never changed back. Sorry for the inconvenience caused. I tested on the saved model and below is the test accuracy (I didn't fix random seed so higher than paper). image

Thanks for help! Now I can get 68.65, approximately equal reported 68.85. Thanks for your nice work again!