search

facebookresearch / habitat-sim

A flexible, high-performance 3D simulator for Embodied AI research.
https://aihabitat.org/
MIT License
2.63k stars 421 forks source link

Patchy semantic map and no depth data #1940

Open pfmephisto opened 1 year ago

pfmephisto commented 1 year ago

Problem

I seem to be getting a patchy segmentation map form my sematic sensor as well as no data form the depth sensor and I am not sure as to why.

image

Habitat-Sim version

➤ conda list | grep habitat
habitat-sim               0.2.2                    pypi_0    pypi
habitat-sim-mutex         1.0              display_bullet    aihabitat
withbullet                2.0                           0    aihabitat

I'm try to using the habitat-matterport-3dresearch example dataset GLAQ4DNUx5U.glb and am running this from inside WSL2.

Any ideas as to what might be causing those issues?

My configs:

sim_settings = {
    "scene": test_scene,  # Scene path
    "scene_dataset": hm3d_scene_dataset,  # the scene dataset configuration files
    "default_agent": 0,
    "sensor_height": 1.5,  # Height of sensors in meters
    "width": 512,  # Spatial resolution of the observations
    "height": 512,
    "hfov": 90,
}

def make_simple_cfg(settings):

    # simulator backend
    sim_cfg = habitat_sim.SimulatorConfiguration()
    sim_cfg.gpu_device_id = 0
    sim_cfg.create_renderer = True
    sim_cfg.enable_physics = False
    sim_cfg.scene_id = settings["scene"]
    sim_cfg.scene_dataset_config_file = settings["scene_dataset"]
    sim_cfg.load_semantic_mesh = True
    sim_cfg.requires_textures = True

    # agent
    agent_cfg = habitat_sim.agent.AgentConfiguration()

    sensor_position = [0.0, settings["sensor_height"], 0.0]
    sensor_resolution = [settings["height"], settings["width"]]

    color_sensor_spec  = habitat_sim.CameraSensorSpec()
    color_sensor_spec.uuid = "color_sensor"
    color_sensor_spec.hfov = settings["hfov"]
    color_sensor_spec.sensor_type = habitat_sim.SensorType.COLOR
    color_sensor_spec.resolution = sensor_resolution
    color_sensor_spec.position = sensor_position
    color_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE
    color_sensor_spec.noise_model = "None"

    depth_sensor_spec = habitat_sim.CameraSensorSpec()
    depth_sensor_spec.uuid = "depth_sensor"
    depth_sensor_spec.hfov = settings["hfov"]
    depth_sensor_spec.sensor_type = habitat_sim.SensorType.DEPTH
    depth_sensor_spec.resolution = sensor_resolution
    depth_sensor_spec.position = sensor_position
    depth_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE
    depth_sensor_spec.noise_model = "None"

    semantic_sensor_spec = habitat_sim.CameraSensorSpec()
    semantic_sensor_spec.uuid = "semantic_sensor"
    semantic_sensor_spec.hfov = settings["hfov"]
    semantic_sensor_spec.sensor_type = habitat_sim.SensorType.SEMANTIC
    semantic_sensor_spec.resolution = sensor_resolution
    semantic_sensor_spec.position = sensor_position
    semantic_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE
    semantic_sensor_spec.noise_model = "None"

    agent_cfg.sensor_specifications = [
        color_sensor_spec,
        depth_sensor_spec,
        semantic_sensor_spec
    ]

    rotation = 60.0
    agent_cfg.action_space = {
        "move_forward": habitat_sim.agent.ActionSpec(
            "move_forward", habitat_sim.agent.ActuationSpec(amount=0.25)
        ),
        "turn_left": habitat_sim.agent.ActionSpec(
            "turn_left", habitat_sim.agent.ActuationSpec(amount=rotation)
        ),
        "turn_right": habitat_sim.agent.ActionSpec(
            "turn_right", habitat_sim.agent.ActuationSpec(amount=rotation)
        ),
    }

    return habitat_sim.Configuration(sim_cfg, [agent_cfg])

cfg = make_simple_cfg(sim_settings)
sim = habitat_sim.Simulator(cfg)
Log while initialising the Simulator

```python [10:34:29:744511]:[Physics] PhysicsManager.cpp(50)::~PhysicsManager : Deconstructing PhysicsManager [10:34:29:744868]:[Scene] SceneManager.h(25)::~SceneManager : Deconstructing SceneManager [10:34:29:744887]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph [10:34:29:759555]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor [10:34:29:759644]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor [10:34:29:760345]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor [10:34:29:760426]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph [10:34:29:760586]:[Scene] SemanticScene.h(47)::~SemanticScene : Deconstructing SemanticScene [10:34:29:970597]:[Gfx] Renderer.cpp(72)::~Impl : Deconstructing Renderer [10:34:29:970930]:[Gfx] WindowlessContext.h(17)::~WindowlessContext : Deconstructing WindowlessContext [10:34:30:230775]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : : Proposing JSON name : default.scene_dataset_config.json from original name : default| This file does not exist. [10:34:30:230883]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DSolid:capsule3DSolid_hemiRings_4_cylRings_1_segments_12_halfLen_0.75_useTexCoords_false_useTangents_false) created and registered. [10:34:30:230915]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DWireframe:capsule3DWireframe_hemiRings_8_cylRings_1_segments_16_halfLen_1) created and registered. [10:34:30:230939]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneSolid:coneSolid_segments_12_halfLen_1.25_rings_1_useTexCoords_false_useTangents_false_capEnd_true) created and registered. [10:34:30:230958]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneWireframe:coneWireframe_segments_32_halfLen_1.25) created and registered. [10:34:30:230975]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeSolid:cubeSolid) created and registered. [10:34:30:230992]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeWireframe:cubeWireframe) created and registered. [10:34:30:231011]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderSolid:cylinderSolid_rings_1_segments_12_halfLen_1_useTexCoords_false_useTangents_false_capEnds_true) created and registered. [10:34:30:231029]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderWireframe:cylinderWireframe_rings_1_segments_32_halfLen_1) created and registered. [10:34:30:231050]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereSolid:icosphereSolid_subdivs_1) created and registered. [10:34:30:231067]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereWireframe:icosphereWireframe_subdivs_1) created and registered. [10:34:30:231085]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereSolid:uvSphereSolid_rings_8_segments_16_useTexCoords_false_useTangents_false) created and registered. [10:34:30:231104]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereWireframe:uvSphereWireframe_rings_16_segments_32) created and registered. [10:34:30:231115]:[Metadata] AssetAttributesManager.cpp(110)::AssetAttributesManager : Built default primitive asset templates : 12 [10:34:30:231805]:[Metadata] SceneDatasetAttributesManager.cpp(35)::createObject : File (default) not found, so new default dataset attributes created and registered. [10:34:30:231821]:[Metadata] MetadataMediator.cpp(120)::createSceneDataset : Dataset default successfully created. [10:34:30:231833]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : : Proposing JSON name : ./data/default.physics_config.json from original name : ./data/default.physics_config.json| This file does not exist. [10:34:30:231850]:[Metadata] PhysicsAttributesManager.cpp(26)::createObject : File (./data/default.physics_config.json) not found, so new default physics manager attributes created and registered. [10:34:30:231859]:[Metadata] MetadataMediator.cpp(210)::setActiveSceneDatasetName : Attempting to create new dataset /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json [10:34:30:231871]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : : Proposing JSON name : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json from original name : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json| This file exists. [10:34:30:231947]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DSolid:capsule3DSolid_hemiRings_4_cylRings_1_segments_12_halfLen_0.75_useTexCoords_false_useTangents_false) created and registered. [10:34:30:231973]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DWireframe:capsule3DWireframe_hemiRings_8_cylRings_1_segments_16_halfLen_1) created and registered. [10:34:30:231996]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneSolid:coneSolid_segments_12_halfLen_1.25_rings_1_useTexCoords_false_useTangents_false_capEnd_true) created and registered. [10:34:30:232015]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneWireframe:coneWireframe_segments_32_halfLen_1.25) created and registered. [10:34:30:232033]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeSolid:cubeSolid) created and registered. [10:34:30:232049]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeWireframe:cubeWireframe) created and registered. [10:34:30:232069]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderSolid:cylinderSolid_rings_1_segments_12_halfLen_1_useTexCoords_false_useTangents_false_capEnds_true) created and registered. [10:34:30:232088]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderWireframe:cylinderWireframe_rings_1_segments_32_halfLen_1) created and registered. [10:34:30:232106]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereSolid:icosphereSolid_subdivs_1) created and registered. [10:34:30:232123]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereWireframe:icosphereWireframe_subdivs_1) created and registered. [10:34:30:232140]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereSolid:uvSphereSolid_rings_8_segments_16_useTexCoords_false_useTangents_false) created and registered. [10:34:30:232157]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereWireframe:uvSphereWireframe_rings_16_segments_32) created and registered. [10:34:30:232169]:[Metadata] AssetAttributesManager.cpp(110)::AssetAttributesManager : Built default primitive asset templates : 12 [10:34:30:232510]:[Metadata] AttributesManagerBase.h(265)::loadAllFileBasedTemplates : Loading 1 Stage Template templates found in /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U [10:34:30:232529]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : : Proposing JSON name : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.basis.stage_config.json from original name : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.basis.glb| This file does not exist. [10:34:30:232567]:[Metadata] AbstractObjectAttributesManagerBase.h(184)::createObject : File (/home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.basis.glb) exists but is not a recognized config filename extension, so new default Stage Template attributes created and registered. [10:34:30:232577]:[Metadata] AttributesManagerBase.h(285)::loadAllFileBasedTemplates : : Loaded file-based templates: 1 [10:34:30:232586]:[Metadata] AttributesManagerBase.h(360)::buildAttrSrcPathsFromJSONAndLoad : :1paths specified in JSON doc forStage Templatetemplates. [10:34:30:232686]:[Metadata] AttributesManagerBase.h(360)::buildAttrSrcPathsFromJSONAndLoad : :1paths specified in JSON doc forScene Instancetemplates. [10:34:30:232700]:[Metadata] SceneDatasetAttributesManager.cpp(35)::createObject : JSON Configuration File (/home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json) based dataset attributes created and registered. [10:34:30:232716]:[Metadata] MetadataMediator.cpp(120)::createSceneDataset : Dataset /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json successfully created. [10:34:30:232725]:[Metadata] MetadataMediator.cpp(217)::setActiveSceneDatasetName : Attempt to create new dataset /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json succeeded. Currently active dataset : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json [10:34:30:232735]:[Metadata] MetadataMediator.cpp(175)::setCurrPhysicsAttributesHandle : Old physics manager attributes changed to ./data/default.physics_config.json successfully. [10:34:30:232745]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb and dataset : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json which is currently active dataset. Renderer: D3D12 (Intel(R) UHD Graphics 630) by Microsoft Corporation OpenGL version: 4.1 (Core Profile) Mesa 22.2.1 Using optional features: GL_ARB_vertex_array_object GL_ARB_ES2_compatibility GL_ARB_separate_shader_objects GL_ARB_robustness GL_ARB_texture_storage GL_ARB_invalidate_subdata GL_ARB_texture_storage_multisample GL_ARB_multi_bind GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_texture_filter_anisotropic GL_KHR_debug Using driver workarounds: no-layout-qualifiers-on-old-glsl mesa-dsa-createquery-except-pipeline-stats mesa-forward-compatible-line-width-range [10:34:30:267102]:[Metadata] MetadataMediator.cpp(297)::getSceneInstanceAttributesByName : Dataset : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json has no preloaded SceneInstanceAttributes or StageAttributes named : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb so loading/creating a new StageAttributes with this name, and then creating a SceneInstanceAttributes with the same name that references this stage. [10:34:30:267150]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : : Proposing JSON name : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.stage_config.json from original name : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb| This file does not exist. [10:34:30:267229]:[Metadata] AbstractObjectAttributesManagerBase.h(184)::createObject : File (/home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb) exists but is not a recognized config filename extension, so new default Stage Template attributes created and registered. [10:34:30:267282]:[Metadata] SceneDatasetAttributes.cpp(47)::addNewSceneInstanceToDataset : Dataset : 'hm3d_annotated_example_basis' : Stage Attributes '/home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb' specified in Scene Attributes exists in dataset library. [10:34:30:267292]:[Metadata] SceneDatasetAttributes.cpp(84)::addNewSceneInstanceToDataset : Dataset : 'hm3d_annotated_example_basis' : Lighting Layout Attributes 'no_lights' specified in Scene Attributes but does not exist in dataset, so creating. [10:34:30:267302]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : : Proposing JSON name : no_lights.lighting_config.json from original name : no_lights| This file does not exist. [10:34:30:267320]:[Metadata] LightLayoutAttributesManager.cpp(36)::createObject : File (no_lights) not found, so new default light layout attributes created and registered. [10:34:30:267332]:[Sim] Simulator.cpp(237)::createSceneInstance : Navmesh file location in scene instance : [10:34:30:267351]:[Scene] SceneGraph.h(85)::createDrawableGroup : Created DrawableGroup: [10:34:30:267360]:[Assets] ResourceManager.cpp(297)::loadSemanticSceneDescriptor : SceneInstance : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb proposed Semantic Scene Descriptor filename : 00861-GLAQ4DNUx5U/GLAQ4DNUx5U_semantic.txt [10:34:30:269534]:[Assets] ResourceManager.cpp(308)::loadSemanticSceneDescriptor : SSD with SceneInstanceAttributes-provided name 00861-GLAQ4DNUx5U/GLAQ4DNUx5U_semantic.txt successfully found and loaded [10:34:30:269564]:[Assets] ResourceManager.cpp(1641)::ConfigureImporterManagerGLExtensions : Importing Basis files as ASTC 4x4. [10:34:30:270843]:[Sim] Simulator.cpp(302)::createSceneInstance : Using scene instance-specified Light key : - no_lights - [10:34:30:270871]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb and dataset : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/hm3d_annotated_example_basis.scene_dataset_config.json which is currently active dataset. [10:34:30:270911]:[Sim] Simulator.cpp(412)::instanceStageForSceneAttributes : Start to load stage named : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb with render asset : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb and collision asset : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb [10:34:30:270997]:[Assets] ResourceManager.cpp(691)::createStageAssetInfosFromAttributes : Frame :{"up":[0,0,1],"front":[0,1,0],"origin":[0,0,0]} for render mesh named : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb|{"up":[0,0,1],"front":[0,1,0],"origin":[0,0,0]} for semantic mesh named : 00861-GLAQ4DNUx5U/GLAQ4DNUx5U_semantic.glb with type specified as semantic|Semantic Txtrs : True [10:34:30:271008]:[Assets] ResourceManager.cpp(431)::loadStage : Loading Semantic Stage mesh : 00861-GLAQ4DNUx5U/GLAQ4DNUx5U_semantic.glb [10:34:30:271017]:[Scene] SceneGraph.h(85)::createDrawableGroup : Created DrawableGroup: [10:34:30:271024]:[Assets] ResourceManager.cpp(973)::loadStageInternal : Attempting to load stage 00861-GLAQ4DNUx5U/GLAQ4DNUx5U_semantic.glb [10:34:30:271032]:[Assets] ResourceManager.cpp(852)::loadRenderAsset : Loading Semantic Mesh asset named: 00861-GLAQ4DNUx5U/GLAQ4DNUx5U_semantic.glb [10:34:30:271040]:[Assets] ResourceManager.cpp(1641)::ConfigureImporterManagerGLExtensions : Importing Basis files as ASTC 4x4. [10:34:30:690635]:[Assets] GenericSemanticMeshData.cpp(211)::buildSemanticMeshData : Parsing Semantic File GLAQ4DNUx5U_semantic.glb w/prim:5 : Inserted Unknown semantic Color #df7afa in map w/ nonSSDObjID = 908 [10:35:00:836403]:[Assets] ResourceManager.cpp(2089)::loadMaterials : Building 57 materials for asset named 'GLAQ4DNUx5U_semantic.glb' : [10:35:01:323934]:[Assets] ResourceManager.cpp(464)::loadStage : Semantic Stage mesh : 00861-GLAQ4DNUx5U/GLAQ4DNUx5U_semantic.glb loaded. [10:35:01:323990]:[Assets] ResourceManager.cpp(507)::loadStage : Start load render asset /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb . [10:35:01:324005]:[Assets] ResourceManager.cpp(973)::loadStageInternal : Attempting to load stage /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb [10:35:01:324032]:[Assets] ResourceManager.cpp(855)::loadRenderAsset : Loading general asset named: /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb [10:35:01:324048]:[Assets] ResourceManager.cpp(1641)::ConfigureImporterManagerGLExtensions : Importing Basis files as ASTC 4x4. [10:35:23:924446]:[Assets] ResourceManager.cpp(2089)::loadMaterials : Building 57 materials for asset named 'GLAQ4DNUx5U.glb' : [10:35:23:924538]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 00:Flat. [10:35:23:924559]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 01:Flat. [10:35:23:924577]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 02:Flat. [10:35:23:924593]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 03:Flat. [10:35:23:924607]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 04:Flat. [10:35:23:924621]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 05:Flat. [10:35:23:924635]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 06:Flat. [10:35:23:924652]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 07:Flat. [10:35:23:924667]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 08:Flat. [10:35:23:924680]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 09:Flat. [10:35:23:924694]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 10:Flat. [10:35:23:924708]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 11:Flat. [10:35:23:924722]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 12:Flat. [10:35:23:924735]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 13:Flat. [10:35:23:924748]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 14:Flat. [10:35:23:924762]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 15:Flat. [10:35:23:924775]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 16:Flat. [10:35:23:924788]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 17:Flat. [10:35:23:924801]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 18:Flat. [10:35:23:924815]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 19:Flat. [10:35:23:924828]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 20:Flat. [10:35:23:924841]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 21:Flat. [10:35:23:924855]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 22:Flat. [10:35:23:924869]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 23:Flat. [10:35:23:924883]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 24:Flat. [10:35:23:924896]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 25:Flat. [10:35:23:924909]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 26:Flat. [10:35:23:924923]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 27:Flat. [10:35:23:924937]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 28:Flat. [10:35:23:924950]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 29:Flat. [10:35:23:924963]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 30:Flat. [10:35:23:924978]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 31:Flat. [10:35:23:924990]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 32:Flat. [10:35:23:925004]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 33:Flat. [10:35:23:925017]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 34:Flat. [10:35:23:925030]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 35:Flat. [10:35:23:925047]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 36:Flat. [10:35:23:925060]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 37:Flat. [10:35:23:925073]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 38:Flat. [10:35:23:925087]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 39:Flat. [10:35:23:925100]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 40:Flat. [10:35:23:925113]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 41:Flat. [10:35:23:925130]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 42:Flat. [10:35:23:925225]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 43:Flat. [10:35:23:925267]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 44:Flat. [10:35:23:925287]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 45:Flat. [10:35:23:925319]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 46:Flat. [10:35:23:925355]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 47:Flat. [10:35:23:925570]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 48:Flat. [10:35:23:925613]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 49:Flat. [10:35:23:925643]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 50:Flat. [10:35:23:925669]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 51:Flat. [10:35:23:925693]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 52:Flat. [10:35:23:925716]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 53:Flat. [10:35:23:925738]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 54:Flat. [10:35:23:925763]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 55:Flat. [10:35:23:925793]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 56:Flat. [10:35:24:534659]:[Sim] Simulator.cpp(439)::instanceStageForSceneAttributes : Successfully loaded stage named : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb [10:35:24:534754]:[Sim] Simulator.cpp(205)::reconfigure : CreateSceneInstance success == true for active scene name : /home/mephisto/repos/habitat-sim/data/habitat-matterport-3dresearch/example/00861-GLAQ4DNUx5U/GLAQ4DNUx5U.glb with renderer. [10:35:24:553539]:[Nav] PathFinder.cpp(410)::build : Building navmesh with 283 x 215 cells [10:35:24:742313]:[Nav] PathFinder.cpp(681)::build : Created navmesh with 566 vertices 273 polygons [10:35:24:742396]:[Sim] Simulator.cpp(835)::recomputeNavMesh : reconstruct navmesh successful ```

aclegg3 commented 1 year ago

Hey @pfmephisto, very odd issue here.

I'm not sure why the depth image is empty. It may need to be normalized to the clipping range (e.g. 10 m).

The semantic issue looks a bit like there is some render clipping bug on your system. My local version works fine on main and I don't see any red flags in these logs.

My first suggestion is try the nightly Habitat conda build or build locally from source on main and ensure you have HM3D Semantics v0.2.

pfmephisto commented 1 year ago

Hey @aclegg3,

Still no luck. With the nightly build running: habitat-viewer /path/to/data/scene_datasets/habitat-test-scenes/skokloster-castle.glb

give me a black screen with updating stats but now image.

Otherwise running my scrip results in an unable to find EGL error although I did not install it with headless.

Platform::WindowlessEglApplication::tryCreateContext(): unable to find EGL device for CUDA device 0     │
WindowlessContext: Unable to create windowless context

Would there be an issue with my graphic card? I am using a NVIDIA Quadro P4200 Mobile

Or does the Linux distro matter? I am on Arch

pfmephisto commented 1 year ago

Running the docker container on windows gives me the same error. Should I be reinstalling my windows drivers for Nvidia/CUDA? I had GPU support previously working inside WSL for pytorch in a different python environment. So I didn't think that the issue might be the underlying windows system.

Command: examples/example_pointnav.py Error:

[09:38:33:018298]:[Metadata] PhysicsAttributesManager.cpp(26)::createObject : JSON Configuration File (./data/default.physics_config.json) based physics manager attributes created and registered.
[09:38:33:018336]:[Metadata] MetadataMediator.cpp(203)::setActiveSceneDatasetName : Previous active dataset  changed to default successfully.
[09:38:33:018374]:[Metadata] MetadataMediator.cpp(175)::setCurrPhysicsAttributesHandle : Old physics manager attributes  changed to ./data/default.physics_config.json successfully.
[09:38:33:018391]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb and dataset : default which is currently active dataset.
Platform::WindowlessEglApplication::tryCreateContext(): unable to find EGL device for CUDA device 0
WindowlessContext: Unable to create windowless context