search

DavidYanAnDe / ARC

MIT License
32 stars 1 forks source link

Inconsistency between the reproduced results and paper's results. #2

Open 2catycm opened 2 months ago

2catycm commented 2 months ago

I followed your instructions in README.md. I prepare the python env with your requirements.txt. Then the vtab-1k is downloaded from the provided onedrive link. (Thanks for your onedrive sharing, saving lots of tedious work on processing vtab) ViT-B_16.npz is also downloaded. Then with the command

python vtab_ARC_train.py --data_path /my/path/to/vtab

which default trains vit/b with ARC method on dataset kitti, yields a extremely low result.

08/21/2024 13:41:20 - INFO - __main__ - Validation Results
08/21/2024 13:41:20 - INFO - __main__ - Valid Loss: 1.99530
08/21/2024 13:41:20 - INFO - __main__ - Valid Accuracy: 0.53727
Training (3168 / 3200 Steps) (loss=0.30644): 100%|-| 32/32 [00:09<00:00,  3.24it/s]
Training (3199 / 3200 Steps) (loss=0.20717):  94%|-| 30/32 [00:04<00:00,  7.94it/s]08/21/2024 13:41:24 - INFO - __main__ - ***** Running Validation *****
08/21/2024 13:41:24 - INFO - __main__ -   Num steps = 3
08/21/2024 13:41:24 - INFO - __main__ -   Batch size = 256
Validating... (loss=2.12428): 100%|-| 3/3 [00:05<00:00,  1.82s/it]
08/21/2024 13:41:30 - INFO - __main__ - 3 [00:05<00:00,  1.29s/it]

08/21/2024 13:41:30 - INFO - __main__ - Validation Results
08/21/2024 13:41:30 - INFO - __main__ - Valid Loss: 1.99528
08/21/2024 13:41:30 - INFO - __main__ - Valid Accuracy: 0.53727
Training (3199 / 3200 Steps) (loss=0.20717):  94%|-| 30/32 [00:09<00:00,  3.04it/s]
08/21/2024 13:41:30 - INFO - __main__ - Best Accuracy:  0.571027
08/21/2024 13:41:30 - INFO - __main__ - End Training!

My GPU is a single 4090.

In the ARC paper table 2, it should be 81.6, but now it is only 57.1!

2catycm commented 2 months ago

I also tried 

python vtab_ARC_train.py --data_path /my/path/to/vtab --dataset cifar

which is even worse! 

08/21/2024 15:00:49 - INFO - __main__ - Validation Results
08/21/2024 15:00:49 - INFO - __main__ - Valid Loss: 2.69603
08/21/2024 15:00:49 - INFO - __main__ - Valid Accuracy: 0.40630
Training (3199 / 3200 Steps) (loss=0.00414):  94%|-| 30/32 [00:18<00:01,  1.66it/s]
08/21/2024 15:00:49 - INFO - __main__ - Best Accuracy:  0.407300
08/21/2024 15:00:49 - INFO - __main__ - End Training!

In the ARC paper table 2, it should be 72.2, but now it is only 40.7!