search

thuml / LogME

Code release for "LogME: Practical Assessment of Pre-trained Models for Transfer Learning" (ICML 2021) and Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs (JMLR 2022)
MIT License
203 stars 18 forks source link

A question about reproducing the results #12

Closed zhangyikaii closed 2 years ago

zhangyikaii commented 2 years ago

Hi, thanks for your excellent work and friendly code style. I encountered some difficulties in reproducing the results.

I directly passed the path where the downloaded dataset was located, but the result was quite different.

Next, I use the dataloader implemented here for the FGVCAircraft dataset. Which class type of this dataset should I use? Different parts of this dataset have a significant impact on the results. So I can not reproduce the results in the paper.

Thanks again. I look forward to hearing from you!

image

image

youkaichao commented 2 years ago

You should use the "variant" type with 100 classes. Here are the 100 classes in my computer:

[image: image.png]

Yi-Kai Zhang @.***> 于2022年7月20日周三 23:49写道：

Hi, thanks for your excellent work and friendly code style. I encountered some difficulties in reproducing the results.

I directly passed the path where the downloaded dataset was located, but the result was quite different.

Next, I use the dataloader implemented here https://github.com/lvyilin/pytorch-fgvc-dataset/blob/master/aircraft.py for the FGVCAircraft dataset. Which class type of this dataset should I use? Different parts of this dataset have a significant impact on the results. So I can not reproduce the results in the paper.

Thanks again. I look forward to hearing from you!

[image: image] https://user-images.githubusercontent.com/46623714/180023729-268ec91c-0a9f-4289-b07f-74ab102b2146.png

[image: image] https://user-images.githubusercontent.com/46623714/180026099-a6ab1ba1-1f07-4037-9d54-de550c638b1e.png

— Reply to this email directly, view it on GitHub https://github.com/thuml/LogME/issues/12, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRJAHQCUUQH7YPQDQTSRBLVVANYVANCNFSM54EGA4IA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- All my best wishes!

KaiChao YOU School of Software Tsinghua University Beijing 100084 People's Republic of China E-mail: @. / @. Homepage: https://youkaichao.github.io

zhangyikaii commented 2 years ago

@youkaichao Thanks for your reply. I found that I also need to take the trainval part. I have reproduced the original result.

Thank you very much!