search

djukicn / loca

LOCA - A Low-Shot Object Counting Network With Iterative Prototype Adaptation (ICCV 2023)
MIT License
44 stars 4 forks source link

Cannot replicate the results for loca_few_shot.pth model checkpoint #3

Open YaelGrins opened 11 months ago

YaelGrins commented 11 months ago

Hi, Thank you very much for sharing your work. I am experiencing an issue where I cannot replicate the results published while using the loca_few_shot.pth model checkpoint file. The results I observed after running the evaluation script are as follows: Val set MAE: 62.53 RMSE: 127.91 Test set MAE: 58.45 RMSE: 149.80 Thanks in advance for your time and assistance.

Best regards, Yael

djukicn commented 11 months ago

@YaelGrins Thank you for your question. Is your issue similar to #2, i.e., did you maybe have issues with loading the pretrained model at first?

niki-amini-naieni commented 9 months ago

These are the results I got for FSC-147 -

From running the evaluation code with the provided checkpoints: Val MAE: 10.22, Val RMSE 32.52, Test MAE: 10.71, Test RMSE: 56.28

From training the model using the provided training code: Val MAE: 10.25, Val RMSE: 34.88, Test MAE: 11.43, Test RMSE: 69.84

Most results (except test RMSE) look reasonable compared to the published ones. The test RMSE from running the training code is quite high but still SOTA for class-agnostic counting.