harveyhuang18 / EMR_Merging

[NeurIPS 2024 Spotlight] EMR-Merging: Tuning-Free High-Performance Model Merging
27 stars 3 forks source link

For Beit3 Merging #4

Closed Zhengsh123 closed 1 week ago

Zhengsh123 commented 1 week ago

I'm sorry, I encountered some unresolved issues during the reproduction process. When reproducing the results of Beit3, I tested on the COCO retrieval task using the default parameters, and the results were as follows.

 Eval result = {"tr_r10": 60.71999669075012, "tr_r5": 49.379998445510864, "tr_r1": 24.059998989105225, "ir_r10": 27.48100757598877, "ir_r5": 19.044382870197296, "ir_r1": 6.89324289560318, "average_score": 31.263105074564617}
Accuracy of the network on the 25010 test images: 31.263%

The results for testing the fine-tuned model are as follows.

* Eval result = {"tr_r10": 97.2000002861023, "tr_r5": 94.37999725341797, "tr_r1": 79.0399968624115, "ir_r10": 90.72771072387695, "ir_r5": 84.60615873336792, "ir_r1": 61.363452672958374, "average_score": 84.55289204915364}
Accuracy of the network on the 25010 test images: 84.553%

Could it be that the default parameters need some adjustments? Also, do the parameters for the other tasks (so far I’ve only tested IN1K, which aligns with the results in the paper) need adjustments as well?

harveyhuang18 commented 1 week ago

Hi,

Thanks for your feedback! I checked the code and found a small bug. We have fixed it. We are sorry for our mistake. Specifically, replace Line #57

param_max  += param_abs#= torch.where(param_abs>param_max, param_abs, param_max)

by:

param_max = torch.where(param_abs>param_max, param_abs, param_max)

Thanks again for pointing this problem out!

Zhengsh123 commented 1 week ago

Thank you for your prompt response. This modification has been very effective.