search

facebookresearch / habitat-lab

A modular high-level library to train embodied AI agents across a variety of tasks and environments.
https://aihabitat.org/
MIT License
1.93k stars 481 forks source link

Missing data for the Social Navigation task #1724

Open aleflabo opened 9 months ago

aleflabo commented 9 months ago

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: 0.3.0 Habitat-Sim: 0.3.0

❓ Questions and Help

I'm trying to run the baseline regarding the Social Navigation task and I have some problems related to the data.

  1. To train the baseline, three datasets are needed: ycb, amazon_berkeley and google_object_dataset. YCB is provided, while the others are not. Where should I find them?
  2. a folder seems missing in the robot folder. Specifically the data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf. Here's the error I have: ... 2023-12-21 16:34:49,209 initializing sim RearrangeSim-v0 2023-12-21 16:34:49,897 Initializing task RearrangePddlSocialNavTask-v0 /home/shiva/miniconda3/envs/habitat/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") Renderer: NVIDIA RTX A6000/PCIe/SSE2 by NVIDIA Corporation OpenGL version: 4.6.0 NVIDIA 535.129.03 Using optional features: GL_ARB_vertex_array_object GL_ARB_separate_shader_objects GL_ARB_robustness GL_ARB_texture_storage GL_ARB_texture_view GL_ARB_framebuffer_no_attachments GL_ARB_invalidate_subdata GL_ARB_texture_storage_multisample GL_ARB_multi_bind GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_texture_filter_anisotropic GL_KHR_debug GL_KHR_parallel_shader_compile GL_NV_depth_buffer_float Using driver workarounds: no-forward-compatible-core-context nv-egl-incorrect-gl11-function-pointers no-layout-qualifiers-on-old-glsl nv-zero-context-profile-mask nv-implementation-color-read-format-dsa-broken nv-cubemap-inconsistent-compressed-image-size nv-cubemap-broken-full-compressed-image-query nv-compressed-block-size-in-bits [16:34:51:027456]:[Warning]:[Sim] Simulator.cpp(533)::instanceStageForSceneAttributes : The active scene does not contain semantic annotations : activeSemanticSceneID_ = 0 2023-12-21 16:34:52,007 Initializing task RearrangePddlSocialNavTask-v0 /home/shiva/miniconda3/envs/habitat/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") /home/shiva/miniconda3/envs/habitat/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") 2023-12-21 16:34:52,757 Number of params to train: 8517019 2023-12-21 16:34:52,758 Agent number of parameters: 8517019 2023-12-21 16:34:52,806 Number of params to train: 0 2023-12-21 16:34:52,806 Agent number of parameters: 0 [16:34:53:030905]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:041604]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:052839]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:053833]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:057524]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:063164]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:067468]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf[16:34:53:071058]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : specifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:080242]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : [16:34:53:080462]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepath[16:34:53:080919]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf specifies the URDF Filepath data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:092970]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:096518]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:099540]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepath[16:34:53:099642]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf specifies the URDF Filepath data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:101073]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named data/robots/hab_spot_arm/urdf/hab_spot_arm.urdfdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. ArticulatedObjectAttributes template named data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf specifies the URDF Filepath data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. [16:34:53:109975]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : [16:34:53:113574]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf specifies the URDF Filepath data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. specifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. ArticulatedObjectAttributes template nameddata/robots/hab_spot_arm/urdf/hab_spot_arm.urdfspecifies the URDF Filepathdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf, but this file cannot be found, so registration is aborted. Error executing job with overrides: [] Traceback (most recent call last): File "/home/aleflabo/habitat-lab/habitat-baselines/habitat_baselines/run.py", line 31, in main execute_exp(cfg, "eval" if cfg.habitat_baselines.evaluate else "train") File "/home/aleflabo/habitat-lab/habitat-baselines/habitat_baselines/run.py", line 60, in execute_exp trainer.train() File "/home/shiva/miniconda3/envs/habitat/lib/python3.9/contextlib.py", line 79, in inner return func(*args, **kwds) File "/home/aleflabo/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/ppo_trainer.py", line 664, in train self._init_train(resume_state) File "/home/aleflabo/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/ppo_trainer.py", line 266, in _init_train observations = self.envs.reset() File "/home/aleflabo/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 389, in reset results.append(read_fn()) File "/home/aleflabo/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 110, in __call__ res = self.read_fn() File "/home/aleflabo/habitat-lab/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 68, in recv buf = self.recv_bytes() File "/home/shiva/miniconda3/envs/habitat/lib/python3.9/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/home/shiva/miniconda3/envs/habitat/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/home/shiva/miniconda3/envs/habitat/lib/python3.9/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. Exception ignored in: <function VectorEnv.__del__ at 0x7f6c1f446ca0> Traceback (most recent call last): File "/home/aleflabo/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 615, in __del__ self.close() File "/home/aleflabo/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 467, in close read_fn() File "/home/aleflabo/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 110, in __call__ res = self.read_fn() File "/home/aleflabo/habitat-lab/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 68, in recv buf = self.recv_bytes() File "/home/shiva/miniconda3/envs/habitat/lib/python3.9/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/home/shiva/miniconda3/envs/habitat/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/home/shiva/miniconda3/envs/habitat/lib/python3.9/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError:
gchhablani commented 8 months ago

Facing the same error:

[11:45:08:624863]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : [11:45:08:624875]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : [11:45:08:624863]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named `ArticulatedObjectAttributes template named `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdfdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf` specifies the URDF Filepath `` specifies the URDF Filepath `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdfdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf`, but this file cannot be found, so registration is aborted.`, but this file cannot be found, so registration is aborted.

[11:45:08:624960]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf` specifies the URDF Filepath `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf`, but this file cannot be found, so registration is aborted.
rvalienter90 commented 7 months ago

Facing the same error:

[11:45:08:624863]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : [11:45:08:624875]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : [11:45:08:624863]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named `ArticulatedObjectAttributes template named `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdfdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf` specifies the URDF Filepath `` specifies the URDF Filepath `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdfdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf`, but this file cannot be found, so registration is aborted.`, but this file cannot be found, so registration is aborted.

[11:45:08:624960]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf` specifies the URDF Filepath `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf`, but this file cannot be found, so registration is aborted.

Did you try this: python -m habitat_sim.utils.datasets_download --uids hab_spot_arm --data-path data/

gchhablani commented 7 months ago

@rvalienter90 Thanks for sharing the command. While this fixed the missing object issue, I get the following error now:

  File "/coc/testnvme/gchhablani3/spring_2024/habitat-lab/habitat-lab/habitat/tasks/rearrange/rearrange_sim.py", line 333, in reconfigure
    self._add_objs(ep_info, should_add_objects, new_scene)
  File "/coc/testnvme/gchhablani3/spring_2024/habitat-lab/habitat-lab/habitat/tasks/rearrange/utils.py", line 718, in wrapper
    ret = f(self, *args, **kwargs)
  File "/coc/testnvme/gchhablani3/spring_2024/habitat-lab/habitat-lab/habitat/tasks/rearrange/rearrange_sim.py", line 614, in _add_objs
    ro.transformation = mn.Matrix4(
AttributeError: 'NoneType' object has no attribute 'transformation'

Also, some of the habitat logs:

[15:01:19:811759]:[Error]:[Core] ManagedContainerBase.h(330)::checkExistsWithMessage : <Object Template>::getObjectCopyByHandle:Object Template managed object handle `data/objects/google_object_dataset/configs/002_master_chef_can.object_config.json` not found in ManagedContainer, so aborting.
[15:01:19:811800]:[Error]:[Physics] PhysicsManager.cpp(92)::addObject : Object creation failed due to unknown attributes data/objects/google_object_dataset/configs/002_master_chef_can.object_config.json
[15:01:19:811812]:[Error]:[Core] ManagedContainerBase.h(205)::getObjectHandleByID : Unknown RigidObject managed object ID: -1 , so unable to retrieve handle.
gchhablani commented 7 months ago

It seems that these objects are not automatically downloaded with the download command:

    additional_object_paths:
      - "data/objects/ycb/configs/"
      - "data/objects/amazon_berkeley/configs/"
      - "data/objects/google_object_dataset/configs/"
gchhablani commented 7 months ago

Overriding the above in the configuration to empty list leads to:

[ERROR,rearrange_task] 2024-02-22 16:15:09,340 pddl_sim_state.py:217 Failed to place the robot.
vaibhavoutat commented 7 months ago

Facing the same error:

[11:45:08:624863]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : [11:45:08:624875]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : [11:45:08:624863]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named `ArticulatedObjectAttributes template named `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdfdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf` specifies the URDF Filepath `` specifies the URDF Filepath `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdfdata/robots/hab_spot_arm/urdf/hab_spot_arm.urdf`, but this file cannot be found, so registration is aborted.`, but this file cannot be found, so registration is aborted.

[11:45:08:624960]:[Error]:[Metadata] AOAttributesManager.cpp(190)::registerObjectFinalize : ArticulatedObjectAttributes template named `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf` specifies the URDF Filepath `data/robots/hab_spot_arm/urdf/hab_spot_arm.urdf`, but this file cannot be found, so registration is aborted.

Did you try this: python -m habitat_sim.utils.datasets_download --uids hab_spot_arm --data-path data/

@rvalienter90 Your command did download the dataset for the hab_spot_arm. But after that I get the following warning and errors.

[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render asset data/humanoids/humanoid_data/female_2/female_2.glb with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file.

[ERROR,rearrange_task] 2024-02-28 12:44:52,700 pddl_sim_state.py:217 Failed to place the robot. [ERROR,rearrange_task] 2024-02-28 12:44:53,053 pddl_sim_state.py:217 Failed to place the robot.

I downloaded the additional object dataset from https://huggingface.co/datasets/ai-habitat/OVMM_objects as suggested by https://github.com/facebookresearch/habitat-lab/issues/1694

And as suggested by the log please ensure 'requires lighting' is enabled in object config file. I enabled all the requires lighting to true , using a python script.(google objects had it false , rest two were defaulting to true) Kindly have a look.

vaibhavoutat commented 7 months ago

I was reading the Habitat 3.0 paper and in the section E (Scenes) they have mentioned.

For our experiments,we use a subset of 59 scenes and limit our objects to the YCB dataset (Calli et al., 2017) for simplicity. So do we actually need other data objects for reproducing the baseline?

aclegg3 commented 7 months ago

@vaibhavoutat we used YCB from https://huggingface.co/datasets/ai-habitat/ycb as the clutter and pick-able objects for rearrangement which were inserted into HSSD scenes. The configs for generating episodes show which objects we used. You could use any objects you want, including the larger set from https://huggingface.co/datasets/ai-habitat/OVMM_objects if you want more diversity, but you only need HSSD and YCB to reproduce the baseline.

gchhablani commented 7 months ago

Thanks @aclegg3 for sharing the URLs to the datasets!

@vaibhavoutat I think the error: Failed to place the robot is not a constant error, but for certain target positions, which may be a bug in placement code or something else.

vaibhavoutat commented 6 months ago

@vaibhavoutat we used YCB from https://huggingface.co/datasets/ai-habitat/ycb as the clutter and pick-able objects for rearrangement which were inserted into HSSD scenes. The configs for generating episodes show which objects we used. You could use any objects you want, including the larger set from https://huggingface.co/datasets/ai-habitat/OVMM_objects if you want more diversity, but you only need HSSD and YCB to reproduce the baseline.

Thanks @aclegg3 will try to reproduce the baseline.

vaibhavoutat commented 6 months ago

Thanks @aclegg3 for sharing the URLs to the datasets!

@vaibhavoutat I think the error: Failed to place the robot is not a constant error, but for certain target positions, which may be a bug in placement code or something else.

@gchhablani Yes it seems so.