Incorrect ground truth semantic outputs

[verityw]

Habitat-Sim version

v0.2.1

🐛 Bug

The ground truth semantic camera outputs in Matterport3D environments seem to not match the environment being observed by the RGB and depth cameras. This occurs even when I use the Colab notebook ECCV 2020: Navigation in this repo and listed at https://aihabitat.org/tutorial/2020/.

Steps to Reproduce

On Colab:

1. Open the notebook: https://colab.research.google.com/github/facebookresearch/habitat-sim/blob/main/examples/tutorials/colabs/ECCV_2020_Navigation.ipynb
2. Comment out the following lines at the end of the Define Observation Display Utility Function cell, since the import throws an error:

   if display:
   from habitat.utils.visualizations import maps

3. Run all cells in sequence up until and not including the Working with the NavMesh Cell. This produces the following outputs, where the semantic image does not correspond with the RGB and depth ones (but it does seem to be executing the correct actions).

Locally: I copied and ran parts of the above notebook example into a .py script and ran it. The following code (saved as a .txt) saves the generated images to a directory called ./example_imgs: test.txt

This outputs the same thing as the Colab notebook. I tried a few other environments from MP3D and they all had similar errors, where the semantic image seems to not match the RGB and depth.

See additional context for more details.

Expected behavior

The semantic image should match that of the RGB and depth images -- see the below image, taken from https://youtu.be/kunFMRJAu2U?t=883, which ran the exact notebook I ran, with no differences.

Additional context

I installed habitat-sim's challenge-2022-branch, but wanted to enable semantics for testing. This page https://github.com/facebookresearch/habitat-sim/blob/main/DATASETS.md says that I need to add a scene config file to the MP3D directory and to specify the path to it in the sim settings dictionary, which I have. However, even without the line:

"scene_dataset":
    "./habitat-challenge-data/data/scene_datasets/mp3d/v1/tasks/mp3d/mp3d.scene_dataset_config.json",

in the dictionary, everything works the same.

[verityw]

Resolved! I needed to add the following line to make_cfgs():

sim_cfg.scene_dataset_config_file = settings["scene_dataset"]

Else, the scene dataset config file path in the dictionary would not do anything.

I am leaving this topic open so that this bug can also be fixed in the notebooks in the repo though.

[aclegg3]

@verityw thanks for reporting this and self-identifying the issue! I don't think the colab was updated with the new config. I'll link the fix PR once it's updated.

[Star-down]

@verityw hello,i dont have mp3d.scene_dataset_config.json file,is it download with mp3d dataset?