search

nik-dim / tall_masks

Official repository of "Localizing Task Information for Improved Model Merging and Compression" [ICML 2024]
https://tall-masks.github.io
MIT License
28 stars 0 forks source link

DTD test set performance #1

Open kasurashan opened 5 days ago

kasurashan commented 5 days ago

Hello.

Thank you for sharing this interesting research and the codes.

https://github.com/nik-dim/tall_masks/blob/db2488ad4a4f1151b096ad72019a6582db1f60fc/results/single_task/ViT-B-32/nonlinear_ft_accuracies.json

image

It seems like the individual model's performance on the DTD test set is unexpectedly high. I'm curious as to why these are coming out this way. Could you explain the possible reasons for this?

Thank you.

wang-kee commented 1 day ago

Hi,

Thanks for your interest in our work.

We have noticed that there was a issue in generating the split of DTD test set. We refer to https://github.com/mlfoundations/task_vectors/issues/1 for generating it.

We will update our checkpoints soon with the updated model checkpoint and masks.

Please feel free to let us know if there are anything else we could help you with.