search

changlin31 / BossNAS

(ICCV 2021) BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search
135 stars 20 forks source link

Question about Ranking nats #12

Open pprp opened 2 years ago

pprp commented 2 years ago

after run code: 

cd ranking_nats
python get_model_score_nats.py

I got: 

kendall tau begin
BossNAS: KendalltauResult(correlation=-0.534180602248828, pvalue=0.0)
(-0.7180607093955225, 0.0)
SpearmanrResult(correlation=-0.7341493538551311, pvalue=0.0)
changlin31 commented 2 years ago

This seems a normal result, except the minus. Referring this to @Trent-tangtao , who performed the ranking experiments.

tangtaogo commented 2 years ago

after run code:

cd ranking_nats
python get_model_score_nats.py

I got:

kendall tau begin
BossNAS: KendalltauResult(correlation=-0.534180602248828, pvalue=0.0)
(-0.7180607093955225, 0.0)
SpearmanrResult(correlation=-0.7341493538551311, pvalue=0.0)

I think you means the architecture rating accuracies on cifar100 dataset are shown in Tab. 4. There is no problem that the result of the code is cifar10, you can refer to the result Table 6: Architecture rating accuracy on NATS-Bench SS with CIFAR datasets. The results of cifar100 will be release later if you need.