search

cherubicXN / hawp

Holistically-Attracted Wireframe Parsing [TPAMI'23] & [CVPR' 20]
MIT License
291 stars 51 forks source link

reproduce mAPJ #4

Closed yanconglin closed 4 years ago

yanconglin commented 4 years ago

Hi @cherubicXN ,

Nice work! Currently I am not able to obtain the mAPJ values in the papers using the pretrained model from you. The sAP values are the same as in the table, but the mAPJ is much lower (24.6 vs 31.6 on York). Since the mAPJ script is not included in repo, I used the one from LCNN with minor modification (the junctions are in different formats). I was wondering if you can kindly provide the script for mAPJ evaluation. Still have no idea what is wrong and would like to figue it out. Thanks in advance！

BTW， congrats on your promotion, Dr.Xue.!

cherubicXN commented 4 years ago

Hi @yanconglin,

Thanks very much!

Please see the source code (L40-L76) at https://github.com/zhou13/lcnn/blob/master/eval-mAPJ.py. Specifically, I followed the implementation of L-CNN by using the predicted heatmaps for evaluation. This would make a difference.

yanconglin commented 4 years ago

I did follow LCNN code. Your model outputs both junc_pred and junc_score in the json file, instead of the heatmaps. So I modified the LCNN code the fit your output. Is this the reason? Then I was wondering what the "junc_pred and junc_score" are.

cherubicXN commented 4 years ago

That's the reason, you may need to modify the code to add the jloc_pred and joff_pred (in L127 and L128) to the output dictionary.

Since I found the mAPJ is not as challenging as sAP, I removed the heatmaps of junction predictions.