AlessioSam
commented
8 months ago I have the same issue. Comparing the metrics reported in the paper to those in habitat-baseline, I believe:
- Finding Success (S) -> social_nav_stats.has_found_human
- Finding Success Weighted by Path Steps (SPS) -> social_nav_stats.found_human_rate_over_epi
- Following Rate (F) -> social_nav_stats.found_human_rate_after_encounter_over_epi
- Collision Rate (CR) -> num_agents_collide
In addition to this doubt, I would also like to understand how your last screenshot (from the README.md in https://github.com/facebookresearch/habitat-lab/tree/main/habitat-baselines) relates to Table 3 in the main paper. My opinion is that it corresponds to the experiment "- humanoid GPS" since the agent's GPS is not being used, even though the performances are different.
I look forward to clarification from the authors.
Habitat-Lab and Habitat-Sim versions
Habitat-Lab: master
Habitat-Sim: master
Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release versions of Habitat-Lab and Habitat-Sim? Your question may already be addressed in the latest versions. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.
Master branch contains 'bleeding edge' code and should be used at your own risk.
Docs and Tutorials
Did you read the docs? https://aihabitat.org/docs/habitat-lab/
Did you check out the tutorials? https://aihabitat.org/tutorial/2020/
Perhaps your question is answered there. If not, carry on!
❓ Questions and Help
Dear Habitat Team,
I wanted to express my gratitude for the recent release, as well as for providing exemplary code for evaluation purposes. I do have a couple of questions regarding replicating the results outlined in the paper.
In comparison to the documented results, how can we correlate them with the metrics provided in the
README.md(https://github.com/facebookresearch/habitat-lab/tree/main/habitat-baselines)? Could you please specify which metrics we should prioritize for comparison?Furthermore, I'm interested in understanding how the results shown in the final screenshot compare to those achieved without the inclusion of humanoid GPS or End-to-end RL. Given that the humanoid GPS (
agent_0_nav_goal_sensor) is not enabled by default in the policy's configuration file,hssd_spot_human_social_nav.yaml, I'm curious how this affects the comparison.Thank you for your help!