search

eric-ai-lab / Aerial-Vision-and-Dialog-Navigation

Codebase of ACL 2023 Findings "Aerial Vision-and-Dialog Navigation"
https://sites.google.com/view/aerial-vision-and-dialog/home
39 stars 6 forks source link

About gp and oracle_gp metric #12

Closed qiaogh97 closed 11 months ago

qiaogh97 commented 11 months ago

Dear authors: I find some differences when computing gp and oracle_gp metrics. In line 347 gt_path[-2] is used as the final destination, but when it comes to line 349, gt_path[-1] is used. I think using the gt_path[-1] is reasonable. So is this a bug or is it written that way on purpose?

https://github.com/eric-ai-lab/Aerial-Vision-and-Dialog-Navigation/blob/403ea73edeb2eb294dc4fa2fa36d0816c85b3dc6/src/env.py#L346-L350

yifeisu commented 11 months ago

same question! 🤔🤔🥶🥶

UeFan commented 11 months ago

Thanks for reporting; it is a bug. It should be '-1', I will push the fix shortly. The Eval.ai evaluation code will be updated as well (submissions will be rerun).

UeFan commented 11 months ago

GP metric in the leaderboard will be affected (our main metric SPL and SR will not be affected). The change should be small because the gt_path[-2] and gt_path[-1] usually have limited differences.

Thanks again and I really appreciate your interest in the project, welcome to email me your WeChatID if you like so I can provide further help if needed and even create some collaboration opportunities.