search

sbelharbi / fcam-wsol

Pytorch implementation of F-CAM. Paper: "F-CAM: Full Resolution Class Activation Maps via Guided Parametric Upscaling".
GNU Lesser General Public License v2.1
15 stars 5 forks source link

Unable to Reproduce Paper Results #2

Closed hiranohachiman closed 1 year ago

hiranohachiman commented 1 year ago

Hello,

I recently tried running the code from your repository and found that I couldn't reproduce the results mentioned in the paper and README. Below is the log of the results I obtained:

DLL 2023-09-07 20:48:30.629816 -   1:30:03.507590  REPORT EPOCH/50: split: val/metric loss: None 
DLL 2023-09-07 20:48:30.629827 -   1:30:03.507601  REPORT EPOCH/50: split: val/metric loss: None_best 
DLL 2023-09-07 20:48:30.629838 -   1:30:03.507612  REPORT EPOCH/50: split: val/metric classification: 39.4 
DLL 2023-09-07 20:48:30.629848 -   1:30:03.507622  REPORT EPOCH/50: split: val/metric classification: 39.4_best 
DLL 2023-09-07 20:48:30.629859 -   1:30:03.507633  REPORT EPOCH/50: split: val/metric localization: 75.6 
DLL 2023-09-07 20:48:30.629869 -   1:30:03.507643  REPORT EPOCH/50: split: val/metric localization: 81.6_best 
DLL 2023-09-07 20:48:30.629880 -   1:30:03.507654  REPORT EPOCH/50: split: val/metric localization_IOU_30: 97.8 
DLL 2023-09-07 20:48:30.629890 -   1:30:03.507664  REPORT EPOCH/50: split: val/metric localization_IOU_30: 97.8_best 
DLL 2023-09-07 20:48:30.629902 -   1:30:03.507676  REPORT EPOCH/50: split: val/metric localization_IOU_50: 75.6 
DLL 2023-09-07 20:48:30.629912 -   1:30:03.507686  REPORT EPOCH/50: split: val/metric localization_IOU_50: 81.6_best 
DLL 2023-09-07 20:48:30.629923 -   1:30:03.507697  REPORT EPOCH/50: split: val/metric localization_IOU_70: 28.1 
DLL 2023-09-07 20:48:30.629933 -   1:30:03.507707  REPORT EPOCH/50: split: val/metric localization_IOU_70: 35.9_best 
DLL 2023-09-07 20:48:30.629944 -   1:30:03.507718  REPORT EPOCH/50: split: val/metric top1_loc_30: 38.8 
DLL 2023-09-07 20:48:30.629954 -   1:30:03.507728  REPORT EPOCH/50: split: val/metric top1_loc_30: 38.8_best 
DLL 2023-09-07 20:48:30.629965 -   1:30:03.507739  REPORT EPOCH/50: split: val/metric top1_loc_50: 31.1 
DLL 2023-09-07 20:48:30.629975 -   1:30:03.507749  REPORT EPOCH/50: split: val/metric top1_loc_50: 33.4_best 
DLL 2023-09-07 20:48:30.629986 -   1:30:03.507760  REPORT EPOCH/50: split: val/metric top1_loc_70: 12.1 
DLL 2023-09-07 20:48:30.629997 -   1:30:03.507771  REPORT EPOCH/50: split: val/metric top1_loc_70: 15.6_best 
DLL 2023-09-07 20:48:30.630007 -   1:30:03.507781  REPORT EPOCH/50: split: val/metric top5_loc_30: 74.4 
DLL 2023-09-07 20:48:30.630017 -   1:30:03.507791  REPORT EPOCH/50: split: val/metric top5_loc_30: 74.6_best 
DLL 2023-09-07 20:48:30.630027 -   1:30:03.507801  REPORT EPOCH/50: split: val/metric top5_loc_50: 58.7 
DLL 2023-09-07 20:48:30.630038 -   1:30:03.507812  REPORT EPOCH/50: split: val/metric top5_loc_50: 62.6_best 
DLL 2023-09-07 20:48:30.630048 -   1:30:03.507822  REPORT EPOCH/50: split: val/metric top5_loc_70: 22.2 
DLL 2023-09-07 20:48:30.630058 -   1:30:03.507832  REPORT EPOCH/50: split: val/metric top5_loc_70: 27.8_best

I followed all the steps mentioned in the README accurately, but the results differ from what is mentioned in the paper. Is there something I might be missing, or any specific steps required for reproducing the results?

Thank you in advance.

sbelharbi commented 1 year ago

hi, you are reporting the validset performance. can you please report the testset performance as in the paper?

thanks

hiranohachiman commented 1 year ago

Hello,

Thank you for your prompt response.

I apologize for the oversight. You're right, I reported the performance on the validset. Here are the results for the testset:

================================================================================
                             PERF - CHECKPOINT: LAST 
================================================================================

DLL 2023-09-10 15:09:31.628458 -   2 days, 19:51:04.506232  REPORT EPOCH/50: split: test/metric loss: None 
DLL 2023-09-10 15:09:31.628468 -   2 days, 19:51:04.506242  REPORT EPOCH/50: split: test/metric loss: None_best 
DLL 2023-09-10 15:09:31.628480 -   2 days, 19:51:04.506254  REPORT EPOCH/50: split: test/metric classification: 43.83845357266137 
DLL 2023-09-10 15:09:31.628491 -   2 days, 19:51:04.506265  REPORT EPOCH/50: split: test/metric classification: 43.83845357266137_best 
DLL 2023-09-10 15:09:31.628513 -   2 days, 19:51:04.506287  REPORT EPOCH/50: split: test/metric localization: 80.66965826717293 
DLL 2023-09-10 15:09:31.628524 -   2 days, 19:51:04.506298  REPORT EPOCH/50: split: test/metric localization: 80.66965826717293_best 
DLL 2023-09-10 15:09:31.628535 -   2 days, 19:51:04.506309  REPORT EPOCH/50: split: test/metric localization_IOU_30: 97.46289264756645 
DLL 2023-09-10 15:09:31.628547 -   2 days, 19:51:04.506321  REPORT EPOCH/50: split: test/metric localization_IOU_30: 97.46289264756645_best 
DLL 2023-09-10 15:09:31.628559 -   2 days, 19:51:04.506333  REPORT EPOCH/50: split: test/metric localization_IOU_50: 80.66965826717293 
DLL 2023-09-10 15:09:31.628571 -   2 days, 19:51:04.506345  REPORT EPOCH/50: split: test/metric localization_IOU_50: 80.66965826717293_best 
DLL 2023-09-10 15:09:31.628582 -   2 days, 19:51:04.506356  REPORT EPOCH/50: split: test/metric localization_IOU_70: 33.39661719019676 
DLL 2023-09-10 15:09:31.628594 -   2 days, 19:51:04.506368  REPORT EPOCH/50: split: test/metric localization_IOU_70: 33.39661719019676_best 
DLL 2023-09-10 15:09:31.628605 -   2 days, 19:51:04.506379  REPORT EPOCH/50: split: test/metric top1_loc_30: 43.02726958923024 
DLL 2023-09-10 15:09:31.628617 -   2 days, 19:51:04.506391  REPORT EPOCH/50: split: test/metric top1_loc_30: 43.02726958923024_best 
DLL 2023-09-10 15:09:31.628628 -   2 days, 19:51:04.506402  REPORT EPOCH/50: split: test/metric top1_loc_50: 36.0717984121505 
DLL 2023-09-10 15:09:31.628639 -   2 days, 19:51:04.506413  REPORT EPOCH/50: split: test/metric top1_loc_50: 36.0717984121505_best 
DLL 2023-09-10 15:09:31.628651 -   2 days, 19:51:04.506425  REPORT EPOCH/50: split: test/metric top1_loc_70: 16.189161201242666 
DLL 2023-09-10 15:09:31.628662 -   2 days, 19:51:04.506436  REPORT EPOCH/50: split: test/metric top1_loc_70: 16.189161201242666_best 
DLL 2023-09-10 15:09:31.628673 -   2 days, 19:51:04.506447  REPORT EPOCH/50: split: test/metric top5_loc_30: 74.97411114946496 
DLL 2023-09-10 15:09:31.628684 -   2 days, 19:51:04.506458  REPORT EPOCH/50: split: test/metric top5_loc_30: 74.97411114946496_best 
DLL 2023-09-10 15:09:31.628700 -   2 days, 19:51:04.506474  REPORT EPOCH/50: split: test/metric top5_loc_50: 62.65101829478771 
DLL 2023-09-10 15:09:31.628711 -   2 days, 19:51:04.506485  REPORT EPOCH/50: split: test/metric top5_loc_50: 62.65101829478771_best 
DLL 2023-09-10 15:09:31.628723 -   2 days, 19:51:04.506497  REPORT EPOCH/50: split: test/metric top5_loc_70: 27.252329996548152 
DLL 2023-09-10 15:09:31.628734 -   2 days, 19:51:04.506508  REPORT EPOCH/50: split: test/metric top5_loc_70: 27.252329996548152_best 
DLL 2023-09-10 15:09:31.630236 -   2 days, 19:51:04.508010  EVAL time TESTSET - CHECKPOINT last : 0:05:31.259713
DLL 2023-09-10 15:09:32.805445 -   2 days, 19:51:05.683219

Regarding the cam-method, I used "CAM over CUB using ResNet50".

Any advice on adjusting the hyperparameters to achieve results closer to those reported in the paper would be greatly appreciated.

Thank you for your assistance.