Questions for gibson-objectnav

[Hoyyyaard]

1. As the Paper says that ZSON use the config A for gibson objectnav dataset. But in “(https://github.com/gunagg/zson/tree/main/scripts)/objnav-eval-v1-gibson.sh” , it uses the config-B model. I am very confuse about this. Can you please inform me some details?
2. As I run the (https://github.com/gunagg/zson/tree/main/scripts)/objnav-eval-v1-gibson.sh, I just get the result success rate much lower than the scores in paper both for config A and B without modifing anything with the setting of SUCCESS DISTANCE 0.1 . Is the success distance for gibson-objectnav 0.1m or 1m likes Zer?

Thank you so much for your answers!

[gunagg]

Yes, it should be config-A for gibson, code has been updated. Thanks.

[Hoyyyaard]

Can you please inform me the success distance of the dataset gibson-objectnav？ Becasue it only get the 20.6% of the sr. I just run the official implement.

[PeihaoChen]

I have the same problem.

When I evaluate zson_conf_A.pth on objnav-gibson dataset and set the success distance to 0.1m using config-A, I get a much lower SR compared to the paper (~20% v.s 31.3%). If I set the success distance to 1m, I get a much higher SR (~35% v.s 31.3%). Are these results reasonable?

In Table 1 of the paper, the authors compared zson with zer. For a fair comparison, the success distance is expected to keep the same as zer (i.e., 1m), and the released model achieves higher performance (~35%) compared to the paper.

[gunagg]

Thanks for bringing to our notice - I have updated the model checkpoint, let me know if you still face issues. As for success_distance - the standard success_distance in habitat is 0.1m (which is a harder task than 1m) and we use that.

[rginjapan]

@PeihaoChen I met the error in the last step, could you plz give me some instructions about installation of slurm? batch: error: s_p_parse_file: unable to status file /etc/slurm-llnl/slurm.conf