How can I try out GOAT without physical robot?

[Welly0902]

Hi, I would like to see how GOAT works using only simulator. Previously, I have added another issue which I still can't solve the error. I would like to ask what specific branch and version will make GOAT works on habitat simulator.

This is what I previously try but failed

• habitat-sim 0.2.3(The simulation gui works properly)
• Follow the steps in home-robot goat branch
• Install habitat-lab branch 018cdd7 under the home-robot dir src/third_party/habitat-lab(Non-interactvie testing works properly)
• Also install Detic and Detectron2 in third_party dir(These also work fine)
• Base on the README in projects/habitat_goat, download the dataset and place it in data/datasets
• Run python projects/habitat_goat/eval_episode.py

The above is what I try. I expect to see the robot move in simulator and do the imagenav demo from GOAT paper. I'm not quite sure GOAT working on habitat-sim (seeing it works on the habitat-sim GUI) is already support or not. Please, give me some hints and kindly point out what I did wrong or expect incorrectly. I would like to make this work and create the documents of how it can be done for anyone who also likes to try this(If this is needed). Thanks a lot!

[yvsriram]

Hi, your setup seems right. The home-robot goat-sim branch should have additional updates. What issue are you facing currently? Is it same as in the other issue?

[yvsriram]

Can you elaborate on what you tested here:

Non-interactvie testing works properly

[Welly0902]

Hi, your setup seems right. The home-robot goat-sim branch should have additional updates. What issue are you facing currently? Is it same as in the other issue?

Hi, thanks for helping! I figured out those errors from hydra are caused by when hydra search for all yaml settings in habitat-lab config. Some of the config paths aren't exist in the config store setting in default_structured_configs.py you mentioned early Because hydra will scan for all config in the config folder, so those configs unrelated to goat will also be request by hydra. Due to the reason, when I use home-robot(goat-sim) with habitat-lab(home-robot_goat_support), it seems some config need to be adjusted/trimmed to resolve the hydra error.

However, I still got two questions:

1. In yaml file /path/to/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/config/habitat/dataset/goat/hm3d.yaml, the dataset data/datasets/goat/hm3d/v0.3/{split}/{split}.json.gz requested can't be find anywhere. The dataset provided by project habitat_goat README doesn't seem right. I try with this dataset and run python projects/habitat_goat/eval_episode.py, but the data structure doesn't fit which caused some errors.
2. When I read paper "Go To Anything", I thought I can see the robot do imagenav task in habitat-sim(with GUI) and generate the 2D semantic map. If I want to see robot runs on habitat-sim GUI doing the GOAT task, which program I should look for?

Thanks again!