search

facebookresearch / habitat-sim

A flexible, high-performance 3D simulator for Embodied AI research.
https://aihabitat.org/
MIT License
2.52k stars 412 forks source link

Question about the semantic in the Image Extractor Tutorial #1825

Open xinchengpang opened 2 years ago

xinchengpang commented 2 years ago

Habitat-Sim version

Habitat-Sim 0.2.2 Habitat-Lab 0.2.2

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-sim/ yes Did you check out the tutorials? https://aihabitat.org/tutorial/2020/ yes

❓ Questions and Help

When I try to run the code in the Image Extractor Tutorial, the semantics are incorrect. The picture below is my result. img-0 img-1 img-2 The results in the tutorial are as follows. expected It looks like my results are not correct.

Log output

[16:19:30:815034]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Dataset>: Proposing JSON name : default.scene_dataset_config.json from original name : default| This file does not exist.
[16:19:30:815098]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DSolid:capsule3DSolid_hemiRings_4_cylRings_1_segments_12_halfLen_0.75_useTexCoords_false_useTangents_false) created and registered.
[16:19:30:815116]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DWireframe:capsule3DWireframe_hemiRings_8_cylRings_1_segments_16_halfLen_1) created and registered.
[16:19:30:815129]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneSolid:coneSolid_segments_12_halfLen_1.25_rings_1_useTexCoords_false_useTangents_false_capEnd_true) created and registered.
[16:19:30:815140]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneWireframe:coneWireframe_segments_32_halfLen_1.25) created and registered.
[16:19:30:815152]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeSolid:cubeSolid) created and registered.
[16:19:30:815161]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeWireframe:cubeWireframe) created and registered.
[16:19:30:815172]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderSolid:cylinderSolid_rings_1_segments_12_halfLen_1_useTexCoords_false_useTangents_false_capEnds_true) created and registered.
[16:19:30:815183]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderWireframe:cylinderWireframe_rings_1_segments_32_halfLen_1) created and registered.
[16:19:30:815194]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereSolid:icosphereSolid_subdivs_1) created and registered.
[16:19:30:815203]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereWireframe:icosphereWireframe_subdivs_1) created and registered.
[16:19:30:815215]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereSolid:uvSphereSolid_rings_8_segments_16_useTexCoords_false_useTangents_false) created and registered.
[16:19:30:815225]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereWireframe:uvSphereWireframe_rings_16_segments_32) created and registered.
[16:19:30:815231]:[Metadata] AssetAttributesManager.cpp(110)::AssetAttributesManager : Built default primitive asset templates : 12
[16:19:30:815365]:[Metadata] SceneDatasetAttributesManager.cpp(35)::createObject : File (default) not found, so new default dataset attributes created  and registered.
[16:19:30:815371]:[Metadata] MetadataMediator.cpp(120)::createSceneDataset : Dataset default successfully created.
[16:19:30:815382]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Physics Manager>: Proposing JSON name : ./data/default.physics_config.json from original name : ./data/default.physics_config.json| This file does not exist.
[16:19:30:815391]:[Metadata] PhysicsAttributesManager.cpp(26)::createObject : File (./data/default.physics_config.json) not found, so new default physics manager attributes created and registered.
[16:19:30:815396]:[Metadata] MetadataMediator.cpp(203)::setActiveSceneDatasetName : Previous active dataset  changed to default successfully.
[16:19:30:815400]:[Metadata] MetadataMediator.cpp(175)::setCurrPhysicsAttributesHandle : Old physics manager attributes  changed to ./data/default.physics_config.json successfully.
[16:19:30:815405]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : data/scene_datasets/habitat-test-scenes/apartment_1.glb and dataset : default which is currently active dataset.
Renderer: Mesa Intel(R) UHD Graphics (TGL GT1) by Intel
OpenGL version: 4.6 (Core Profile) Mesa 22.0.1
Using optional features:
    GL_ARB_vertex_array_object
    GL_ARB_ES2_compatibility
    GL_ARB_separate_shader_objects
    GL_ARB_robustness
    GL_ARB_texture_storage
    GL_ARB_invalidate_subdata
    GL_ARB_texture_storage_multisample
    GL_ARB_multi_bind
    GL_ARB_direct_state_access
    GL_ARB_get_texture_sub_image
    GL_ARB_texture_filter_anisotropic
    GL_KHR_debug
Using driver workarounds:
    no-layout-qualifiers-on-old-glsl
    mesa-implementation-color-read-format-dsa-explicit-binding
    mesa-dsa-createquery-except-pipeline-stats
    mesa-forward-compatible-line-width-range
[16:19:30:897142]:[Metadata] MetadataMediator.cpp(297)::getSceneInstanceAttributesByName : Dataset : default has no preloaded SceneInstanceAttributes or StageAttributes named : data/scene_datasets/habitat-test-scenes/apartment_1.glb so loading/creating a new StageAttributes with this name, and then creating a SceneInstanceAttributes with the same name that references this stage.
[16:19:30:897175]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Stage Template>: Proposing JSON name : data/scene_datasets/habitat-test-scenes/apartment_1.stage_config.json from original name : data/scene_datasets/habitat-test-scenes/apartment_1.glb| This file does not exist.
[16:19:30:897230]:[Metadata] AbstractObjectAttributesManagerBase.h(184)::createObject : File (data/scene_datasets/habitat-test-scenes/apartment_1.glb) exists but is not a recognized config filename extension, so new default Stage Template attributes created and registered.
[16:19:30:897257]:[Metadata] SceneDatasetAttributes.cpp(47)::addNewSceneInstanceToDataset : Dataset : 'default' : Stage Attributes 'data/scene_datasets/habitat-test-scenes/apartment_1.glb' specified in Scene Attributes exists in dataset library.
[16:19:30:897263]:[Metadata] SceneDatasetAttributes.cpp(84)::addNewSceneInstanceToDataset : Dataset : 'default' : Lighting Layout Attributes 'no_lights' specified in Scene Attributes but does not exist in dataset, so creating.
[16:19:30:897271]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Lighting Layout>: Proposing JSON name : no_lights.lighting_config.json from original name : no_lights| This file does not exist.
[16:19:30:897284]:[Metadata] LightLayoutAttributesManager.cpp(36)::createObject : File (no_lights) not found, so new default light layout attributes created and registered.
[16:19:30:897291]:[Sim] Simulator.cpp(237)::createSceneInstance : Navmesh file location in scene instance : data/scene_datasets/habitat-test-scenes/apartment_1.navmesh
[16:19:30:897300]:[Sim] Simulator.cpp(242)::createSceneInstance : Loading navmesh from data/scene_datasets/habitat-test-scenes/apartment_1.navmesh
[16:19:30:897316]:[Nav] PathFinder.cpp(852)::loadNavMesh : NavMeshSettings aren't present, guessing that they are the default
[16:19:30:897360]:[Sim] Simulator.cpp(244)::createSceneInstance : Navmesh Loaded.
[16:19:30:897372]:[Scene] SceneGraph.h(85)::createDrawableGroup : Created DrawableGroup: 
[16:19:30:897379]:[Assets] ResourceManager.cpp(297)::loadSemanticSceneDescriptor : SceneInstance : data/scene_datasets/habitat-test-scenes/apartment_1.glb proposed Semantic Scene Descriptor filename : data/scene_datasets/habitat-test-scenes/apartment_1.scn
[16:19:30:897388]:[Assets] ResourceManager.cpp(344)::loadSemanticSceneDescriptor : SSD File Naming Issue! Neither SceneInstanceAttributes-provided name : data/scene_datasets/habitat-test-scenes/apartment_1.scn  nor constructed filename : data/scene_datasets/habitat-test-scenes/info_semantic.json exist on disk.
[16:19:30:897407]:[Assets] ResourceManager.cpp(1641)::ConfigureImporterManagerGLExtensions : Importing Basis files as ASTC 4x4.
[16:19:30:897901]:[Sim] Simulator.cpp(302)::createSceneInstance : Using scene instance-specified Light key : - no_lights -
[16:19:30:897923]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : data/scene_datasets/habitat-test-scenes/apartment_1.glb and dataset : default which is currently active dataset.
[16:19:30:897942]:[Sim] Simulator.cpp(412)::instanceStageForSceneAttributes : Start to load stage named : data/scene_datasets/habitat-test-scenes/apartment_1.glb with render asset : data/scene_datasets/habitat-test-scenes/apartment_1.glb and collision asset : data/scene_datasets/habitat-test-scenes/apartment_1.glb
[16:19:30:897980]:[Assets] ResourceManager.cpp(691)::createStageAssetInfosFromAttributes : Frame :{"up":[0,0,1],"front":[0,1,0],"origin":[0,0,0]} for render mesh named : data/scene_datasets/habitat-test-scenes/apartment_1.glb|{"up":[0,0,1],"front":[0,1,0],"origin":[0,0,0]} for semantic mesh named :  with type specified as semantic|Semantic Txtrs : False
[16:19:30:897989]:[Assets] ResourceManager.cpp(507)::loadStage : Start load render asset data/scene_datasets/habitat-test-scenes/apartment_1.glb .
[16:19:30:897993]:[Assets] ResourceManager.cpp(973)::loadStageInternal : Attempting to load stage data/scene_datasets/habitat-test-scenes/apartment_1.glb 
[16:19:30:898013]:[Assets] ResourceManager.cpp(855)::loadRenderAsset : Loading general asset named: data/scene_datasets/habitat-test-scenes/apartment_1.glb
[16:19:30:898018]:[Assets] ResourceManager.cpp(1641)::ConfigureImporterManagerGLExtensions : Importing Basis files as ASTC 4x4.
[16:19:31:574567]:[Assets] ResourceManager.cpp(2089)::loadMaterials : Building 1 materials for asset named 'apartment_1.glb' : 
[16:19:31:574630]:[Assets] ResourceManager.cpp(2194)::loadMaterials : Idx 00:Flat.
[16:19:31:647744]:[Sim] Simulator.cpp(439)::instanceStageForSceneAttributes : Successfully loaded stage named : data/scene_datasets/habitat-test-scenes/apartment_1.glb
[16:19:31:647789]:[Sim] Simulator.cpp(471)::instanceStageForSceneAttributes : 
---
The active scene does not contain semantic annotations : activeSemanticSceneID_ = 0  
---
[16:19:31:647807]:[Sim] Simulator.cpp(205)::reconfigure : CreateSceneInstance success == true for active scene name : data/scene_datasets/habitat-test-scenes/apartment_1.glb  with renderer.
/home/pangxincheng/.conda/envs/torch1.8/lib/python3.7/site-packages/habitat_sim/utils/data/pose_extractor.py:59: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array(poses)
[16:19:37:535316]:[Physics] PhysicsManager.cpp(50)::~PhysicsManager : Deconstructing PhysicsManager
[16:19:37:535393]:[Scene] SceneManager.h(25)::~SceneManager : Deconstructing SceneManager
[16:19:37:535403]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph
[16:19:37:535816]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor
[16:19:37:535841]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor
[16:19:37:535855]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor
[16:19:37:535888]:[Scene] SemanticScene.h(47)::~SemanticScene : Deconstructing SemanticScene
[16:19:37:539390]:[Gfx] Renderer.cpp(72)::~Impl : Deconstructing Renderer
[16:19:37:539430]:[Gfx] WindowlessContext.h(17)::~WindowlessContext : Deconstructing WindowlessContext
[16:19:37:542680]:[Sim] Simulator.cpp(66)::~Simulator : Deconstructing Simulator
aclegg3 commented 2 years ago

Hey @xinchengpang, it does seem like this tutorial is slightly outdated.

The scene "data/scene_datasets/habitat-test-scenes/apartment_1.glb" referenced in the tutorial does not come with semantic annotations. There are semantic annotations for this scene packaged with the Replica-Dataset referenced in the 2nd part of the tutorial. You may want to try scenes from MP3D or HM3D semantics. Details on acquiring and using these dataset is available on the github datasets page.

dhruvbatra commented 2 years ago

@mpiseno — can you update this please?

aclegg3 commented 1 year ago

@mpiseno checking in on this. When you have a PR, please link it to this issue.