search

XiaohanLei / IEVE

PyTorch implementation of CVPR 2024 paper: Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation
https://xiaohanlei.github.io/projects/IEVE/
18 stars 2 forks source link

Evaluation quesion #5

Closed Cherryreg closed 1 month ago

Cherryreg commented 1 month ago

Hello author, thank you very much for your outstanding work, which I have benefited a lot from.

I had some problems with replicating, and running python main.py directly, following the README instructions, got much worse results than in the paper. Final InstanceImageNav succ/spl/dtg: 0.46000/0.20507/3.11719(500) According to the presentation in your paper, there are parts of the framework that require a few training. Is it relevant to this, and if so, can you provide the model trained in the paper? also hope that you can provide test log Perhaps there are other details that I did not notice that led to poor test results. I hope you can give me some suggestions. Thank you for taking time out of your busy schedule to reply!

XiaohanLei commented 1 month ago

This is very strange. I will run it myself later and save the log of the run.

XiaohanLei commented 1 month ago

After carefully checking the code, we find a possible problem. In the paper, we report the use of velocity control, and our experiments are also conducted under velocity control. However, when we open source the repository, we use discrete waypoint control for simplicity, and do not perform too many checks when using this action space. We will revert to velocity control, check our code, and then commit a new version of this repository. Thank you for bringing our attention to the issue! If there exists any other issue, feel free to contact us!

XiaohanLei commented 1 month ago

Now the action space aligns with the paper reported (velocity control), the results should be able to align with the paper. We run ~60 episodes for evaluation and the reported success is around 0.73. Close this issue. If you have any other problem, feel free to contact us!