search

facebookresearch / home-robot

Mobile manipulation research tools for roboticists
MIT License
874 stars 121 forks source link

/hssd-hab/hssd-hab-uncluttered.scene_dataset_config.json| This file does not exist #347

Closed FUIGUIMURONG closed 1 year ago

FUIGUIMURONG commented 1 year ago

When I run python projects/habitat_ovmm/eval_baselines_agent.py --env_config projects/habitat_ovmm/configs/env/hssd_demo.yaml There is a error:


/home/sdp/mambaforge/envs/home-robot/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673738523684/work/aten/src/ATen/native/TensorShape.cpp:3190.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
2023-08-28 17:18:19,583 Initializing dataset OVMMDataset-v0
2023-08-28 17:18:20,965 initializing sim OVMMSim-v0
[17:18:21:584739]:[Metadata] StageAttributesManager.cpp(86)::registerObjectFinalize : Render asset template handle : data/hssd-hab/scenes-uncluttered/104348361_171513414.scene_instance.json specified in stage template with handle : data/hssd-hab/scenes-uncluttered/104348361_171513414.scene_instance.json does not correspond to any existing file or primitive render asset.  Aborting.
Traceback (most recent call last):
  File "/home/sdp/home-robot/projects/habitat_ovmm/eval_baselines_agent.py", line 98, in <module>
    metrics = evaluator.evaluate(
  File "/home/sdp/home-robot/projects/habitat_ovmm/evaluator.py", line 458, in evaluate
    self._env = create_ovmm_env_fn(self.config)
  File "/home/sdp/home-robot/projects/habitat_ovmm/utils/env_utils.py", line 34, in create_ovmm_env_fn
    habitat_env = env_class(config=habitat_config, dataset=dataset)
  File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 117, in __init__
    base_env = RLTaskEnv(config=config, dataset=dataset)
  File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 43, in __init__
    super().__init__(config, dataset)
  File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 382, in __init__
    self._env = Env(config, dataset)
  File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 114, in __init__
    self._sim = make_sim(
  File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/sims/registration.py", line 19, in make_sim
    return _sim(**kwargs)
  File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/rearrange_sim.py", line 82, in __init__
    super().__init__(config)
  File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py", line 289, in __init__
    super().__init__(self.sim_config)
  File "<attrs generated init habitat_sim.simulator.Simulator>", line 11, in __init__
  File "/home/sdp/mambaforge/envs/home-robot/lib/python3.9/site-packages/habitat_sim-0.2.4-py3.9-linux-x86_64.egg/habitat_sim/simulator.py", line 118, in __attrs_post_init__
    self.__set_from_config(self.config)
  File "/home/sdp/mambaforge/envs/home-robot/lib/python3.9/site-packages/habitat_sim-0.2.4-py3.9-linux-x86_64.egg/habitat_sim/simulator.py", line 227, in __set_from_config
    self._config_backend(config)
  File "/home/sdp/mambaforge/envs/home-robot/lib/python3.9/site-packages/habitat_sim-0.2.4-py3.9-linux-x86_64.egg/habitat_sim/simulator.py", line 200, in _config_backend
    super().__init__(config.sim_cfg, config.metadata_mediator)
AssertionError: ESP_CHECK failed: Missing (at least) one of scene dataset attributes, stage attributes, or dataset scene attributes for scene 'data/hssd-hab/scenes-uncluttered/104348361_171513414.scene_instance.json'.  Likely an invalid scene name.```
I appriciate if you give me some advice.
yvsriram commented 1 year ago

Does this file exist for you: data/hssd-hab/scenes-uncluttered/104348361_171513414.scene_instance.json? If yes, can you please try: cd data/hssd-hab and git lfs pull.

Did you use the install script at: https://github.com/facebookresearch/home-robot/blob/main/projects/habitat_ovmm/install.sh? If you are using this script, this should already be done for you.

Also, can you share the hssd-hab commit you are using? cd data/hssd-hab and git log.

FUIGUIMURONG commented 1 year ago

Thanks for replying. The file: data/hssd-hab/scenes-uncluttered/104348361_171513414.scene_instance.json does not exist. To be specific, There is something wrong with git on the remote server, so I git clone the hssd-hab on my local computer and uploader the dataset to the remote server. Because the dataset is too large, I create the link in data/hssd-hab that points to the uploaded dataset. 1.As you advise, I just try: cd data/hssd-hab and git lfs pull. Weirdly, it has no response. 2.This is the latest gitlog:

(home-robot) sdp@xdy-Super-Server:~/home-robot/data/hssd-hab$ git log
commit b643698a372c9b64af8995d7295d28ba8b7dfe99 (HEAD -> main, origin/main, origin/HEAD)
Author: Chris Paxton <cpaxton@fb.com>
Date:   Wed Aug 16 13:53:06 2023 -0400

    Updates for fixing head tilt + motion in smaller kitchen (#241)

commit 50780f78a4fb43b482b65ce360afcc2e101af998
Author: Xiaohan Zhang <43056077+keke-220@users.noreply.github.com>
Date:   Wed Aug 16 10:18:52 2023 -0400

    Add quality of life checks to make sure robots are homed and initalized right (#338)

    * Add quality of life checks for experimenters to make sure robots are
    homed and working properly

    * add a note about creating a catkin workspace

    ---------

    Co-authored-by: Chris Paxton <cpaxton@fb.com>

commit 2972734c7b0038ee84ab272e3e0a4454d1cd874f
Author: Sriram Yenamandra <yvsriram99@gmail.com>
Date:   Tue Aug 15 10:22:30 2023 -0400

    minor fix info is not Nnne (#337)

commit 28149f5255224760e9c03248576cd5f433f1510d
Author: Sriram Yenamandra <yvsriram99@gmail.com>
Date:   Mon Aug 14 16:56:16 2023 -0400

    add README for installing perception modules (#315)

    * add README for installing perception modules

    * Update install_deps.sh

    * Mention GroundedSAM in README.md

commit 647f9975121f507dc48913bb8ba129028847e8e9
Author: Ruslan <partsey2412@gmail.com>
Date:   Mon Aug 14 19:28:40 2023 +0300

    added ubuntu20.04 base challenge Docker image; updated challenge readme (#335)

commit 4e3149117a1d237a8d7a35eb2f92dec3145dc0bd
Author: Chris Paxton <cpaxton@fb.com>
Date:   Wed Aug 9 11:07:19 2023 -0400

    Extra note about python (#331)

    * Add extra note about python 3.9

    * extra note about python and dependencies

commit 61743670d0900ef8ce6753dafdd1bd348822e553
Author: Chris Paxton <cpaxton@fb.com>
Date:   Mon Aug 7 15:43:22 2023 -0400

    Fix bug with agent evaluation (#327)

    * add __init__

    * fix agent command because of condition in ovmm perception

commit 90bef70ef8642c601b1e054b6fc5d7da8b6659ff
Author: Chris Paxton <cpaxton@fb.com>
Date:   Mon Aug 7 12:07:00 2023 -0400

    Add specific phase commands since they are misleading (#326)

    * add phase example commands

    * more detailed commands

    * Update the documentation
yvsriram commented 1 year ago

This is git log for home-robot. I wanted to see this for the hssd-hab repo.

Since the file does not exist, most likely you are on the wrong hssd-hab commit. You should use the commits linked from home-robot .gitmodules. I would advise downloading the git submodules locally:

  1. Clone home-robot locally
  2. git submodule update --init data/hssd-hab (or run https://github.com/facebookresearch/home-robot/blob/main/projects/habitat_ovmm/install.sh)
FUIGUIMURONG commented 1 year ago

Thanks for your advice. I want to figure it more clear. 1.I change to the original hssd-hab directory and usegit log(not the linked directory). And it shows like the following. Does the git log for hssd-hab?:

(home-robot) sdp@xdy-Super-Server:/data/sdp/data/OVMM/hssd-hab$ git log
commit 66bc2f1f15e5e7fbf5d8db4ad0458b569fac352d (HEAD -> main, tag: v0.2.4, origin/main, origin/HEAD)
Author: Mukul Khanna <mukul18khanna@gmail.com>
Date:   Wed Aug 16 19:50:40 2023 +0530

    Uploads recompressed object GLBs

commit 5f7980f98a7adddb383e0ef4af8293b08eb0a89c
Author: Mukul Khanna <mukul18khanna@gmail.com>
Date:   Wed Aug 16 13:14:03 2023 +0530

    Adds semantic lexicon and csv file with object metadata

commit 53ade654035c9f165c63ba8496ee82a661c57f67
Author: Mukul Khanna <mukul18khanna@gmail.com>
Date:   Wed Aug 16 13:08:43 2023 +0530

    Adds train/val scene splits yaml

commit 4df914754fccb77a307d69b638434327e7200e9c
Author: Mukul Khanna <mukulkhanna@users.noreply.huggingface.co>
Date:   Mon Aug 14 17:17:47 2023 +0000
  1. Is the data structure for hssd-hab? I used the git clone https://huggingface.co/datasets/hssd/hssd-hab to download thehss-hab locally. Is the right way to download the dataset? 
    (home-robot) sdp@xdy-Super-Server:/data/sdp/data/OVMM/hssd-hab$ ll
总用量 76
drwxrwxr-x  7 sdp sdp  4096 8月  27 22:40 ./
drwxrwxr-x  4 sdp sdp  4096 8月  27 17:25 ../
drwxrwxr-x  8 sdp sdp  4096 8月  27 20:04 .git/
-rw-rw-r--  1 sdp sdp  2462 8月  27 15:56 .gitattributes
-rw-rw-r--  1 sdp sdp    11 8月  27 15:56 .gitignore
-rw-rw-r--  1 sdp sdp   664 8月  27 15:56 hssd-hab.scene_dataset_config.json
drwxrwxr-x 21 sdp sdp  4096 8月  27 22:35 objects/
-rw-rw-r--  1 sdp sdp  3199 8月  27 15:56 README.md
drwxrwxr-x  2 sdp sdp 16384 8月  27 22:40 scenes/
-rw-rw-r--  1 sdp sdp  3884 8月  27 15:57 scene_splits.yaml
drwxrwxr-x  2 sdp sdp  4096 8月  27 22:40 semantics/
drwxrwxr-x  2 sdp sdp 20480 8月  27 22:42 stages/
yvsriram commented 1 year ago
  1. Yes, that is the git log. This is the main branch which does not have uncluttered scenes we are using.
  2. This is actually not the right way as it downloads the main branch. Like before, I would advise you to use submodules and download script (or commands from it) so that you download the right combination of assets. You can run these locally and then upload them if you are having git issues on your server.
FUIGUIMURONG commented 1 year ago
  1. Yes, that is the git log. This is the main branch which does not have uncluttered scenes we are using.
  2. This is actually not the right way as it downloads the main branch. Like before, I would advise you to use submodules and download script (or commands from it) so that you download the right combination of assets. You can run these locally and then upload them if you are having git issues on your server.

Thanks for your advice. I would do as you advise and see the result.

FUIGUIMURONG commented 1 year ago

As you advise, I do the git submodule update --init --recursive , and thegit submodule update --init data/hssd-hab to download the hssd-hab. The similar way to download the dataset /dataset/ovmm and object. When I run thepython projects/habitat_ovmm/eval_baselines_agent.py --env_config projects/habitat_ovmm/configs/env/hssd_demo.yaml, it shows like this: 1.

Details

[18:01:16:329175]:[Sim] Simulator.cpp(932)::recomputeNavMesh : reconstruct navmesh successful Traceback (most recent call last): File "/home/sdp/home-robot/projects/habitat_ovmm/eval_baselines_agent.py", line 97, in metrics = evaluator.evaluate( File "/home/sdp/home-robot/projects/habitat_ovmm/evaluator.py", line 459, in evaluate return self.local_evaluate(agent, num_episodes) File "/home/sdp/home-robot/projects/habitat_ovmm/evaluator.py", line 259, in local_evaluate observations, done = self._env.reset(), False File "/home/sdp/home-robot/src/home_robot_sim/home_robot_sim/env/habitat_ovmm_env/habitat_ovmm_env.py", line 116, in reset habitat_obs = self.habitat_env.reset() File "/home/sdp/mambaforge/envs/home-robot/lib/python3.9/site-packages/gym/core.py", line 283, in reset return self.env.reset(**kwargs) File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/utils/gym_adapter.py", line 300, in reset obs = self._env.reset() File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 54, in reset observations = super().reset() File "/home/sdp/mambaforge/envs/home-robot/lib/python3.9/contextlib.py", line 79, in inner return func(*args, **kwds) File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 422, in reset return self._env.reset() File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 262, in reset observations = self.task.reset(episode=self.current_episode) File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/ovmm/sub_tasks/nav_to_obj_task.py", line 53, in reset obs = super().reset(episode) File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/sub_tasks/nav_to_obj_task.py", line 267, in reset return self._get_observations(episode) File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/rearrange_task.py", line 193, in _get_observations task_obs = self.sensor_suite.get_observations( File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/simulator.py", line 229, in get_observations return Observations(self.sensors, *args, **kwargs) File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/simulator.py", line 123, in __init__ data = [ File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/simulator.py", line 124, in (uuid, sensor.get_observation(*args, **kwargs)) File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/ovmm/sub_tasks/nav_to_obj_sensors.py", line 118, in get_observation obs[..., 1] = self._get_obs_channel( File "/home/sdp/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/ovmm/sub_tasks/nav_to_obj_sensors.py", line 87, in _get_obs_channel handle = self._sim.receptacles[ KeyError: 'receptacle_mesh_1d5a78b46d32bf41584c800a0dfa2536d7f0b395.0000'

  1. I check the git log for the three dataset:

commit 3fc838aa95d14da91ce69fd5243bf38e399ba2a9 Author: Arun Ramachandran arun.jhurricane@gmail.com Date: Mon Jun 19 01:52:36 2023 +0000

initial commit

- As for` hssd-hab`:

```python
(base) sdp@xdy-Super-Server:~/home-robot/data/hssd-hab$ git log
commit 22a8c888524f37d3e11dd8f98a21bcba8cad4e12 (HEAD)
Author: Manolis Savva <manolis.savva@gmail.com>
Date:   Tue Jun 20 08:45:59 2023 -0700

    add receptacle plys

commit d69278fe1247c27411f2605c38bbf0391f34c870
Author: Manolis Savva <manolis.savva@gmail.com>
Date:   Mon Jun 19 16:22:48 2023 -0700

    add colliders and receptacles

commit 7f2834a47e6798593488cfea95b41049e0288b4f
Author: Manolis Savva <manolis.savva@gmail.com>
Date:   Mon Jun 19 13:54:32 2023 -0700

    add uncluttered scenes, navmeshes

commit d8de88bfc22e89ce911f82df994838e5d1815b2e (tag: v0.2.3)
Author: Manolis Savva <manolis.savva@gmail.com>
Date:   Wed Jun 14 20:16:33 2023 -0700

    Update README.md
(base) sdp@xdy-Super-Server:~/home-robot/data/objects$ git log
commit d11b3a0edd401f285d04d9c7bca493f45e0297b8 (HEAD)
Author: Arun Ramachandran <JHurricane96@users.noreply.huggingface.co>
Date:   Wed Jun 21 14:19:04 2023 +0000

    missing_objects (#1)

    - add missing object files (6201f4b61a85a794ecc79acfa1359781cce26114)

    Co-authored-by: sriram <yvsriram@users.noreply.huggingface.co>

commit 1863868d39af7d92f4af46fd899e8bd689c177a4

I cannot locate which dataset miss something resulting KeyError:'receptacle_mesh_1d5a78b46d32bf41584c800a0dfa2536d7f0b395.0000'. I really hope you can give me some advice.

yvsriram commented 1 year ago

Yes, now it looks like the dataset you downloaded is incompatible with the habitat-lab version. The home-robot on your local machine (where you downloaded the data) uses an older home-robot commit. Can you please run git pull && git submodule update --init --recursive locally to download the latest datasets?

FUIGUIMURONG commented 1 year ago

/I run git pull && git submodule update --init --recursive as you suggest , and it shows like this:

Details

```python (base) sdp@xdy-Super-Server:~/home-robot$ git pull && git submodule update --init --recursive remote: Enumerating objects: 426, done. remote: Counting objects: 100% (426/426), done. remote: Compressing objects: 100% (202/202), done. remote: Total 426 (delta 252), reused 371 (delta 218), pack-reused 0 Received objects: 100% (426/426), 6.11 MiB | 27.00 KiB/s, done. Processing deltas: 100% (252/252), completed with 71 local objects. From https://github.com/facebookresearch/home-robot b643698a..f4134052 main -> origin/main 727230fc..04c90b9d alexsax/scannet -> origin/alexsax/scannet * [New branch] alexsax/scannet_data -> origin/alexsax/scannet_data * [New branch] cpaxton/voxel-rrt -> origin/cpaxton/voxel-rrt 4baec203..d210deec goat -> origin/goat * [New branch] mchang/abnb1 -> origin/mchang/abnb1 * [New branch] new_ovmm_episodes -> origin/new_ovmm_episodes 4b021514..2d8a4062 ray_tracing -> origin/ray_tracing * [New lable] v0.1.1 -> v0.1.1 Fetching submodule src/third_party/habitat-lab coming from https://github.com/facebookresearch/habitat-lab 12e735d2..9ccca099 ovmm -> origin/ovmm 1be042fc..9efa20d1 SIRo -> origin/SIRo b2011c70..cafdd508 SIRo_basevelhuman -> origin/SIRo_basevelhuman b4a8bc9a..67a07cef SIRo_dev_add_nn_policy_backup_backup_integrate_hrl_with_ll -> origin/SIRo_dev_add_nn_policy_backup_backup_integrate_hrl_with_ll a60f8a39..1f8224e9 SIRo_dev_add_nn_policy_backup_backup_integrate_hrl_with_ll_integrate_nn_nav -> origin/SIRo_dev_add_nn_policy_backup_backup_integrate_hrl_with_ll_integrate_nn_nav * [New branch] dataset-pickle -> origin/dataset-pickle * [New branch] hab3_cleanup -> origin/hab3_cleanup eb39df34..462c37b6 main -> origin/main 20848ed1..b8694e35 ovmm_main_merge -> origin/ovmm_main_merge 98265ddb..bfc784e5 social_nav_randwalk_integration -> origin/social_nav_randwalk_integration * [New branch] social_nav_randwalk_integration_oracle -> origin/social_nav_randwalk_integration_oracle Updating b643698a..f4134052 Fast-forward .gitignore | 19 +- data/datasets/ovmm | 2 +- data/hssd-hab | 2 +- projects/habitat_ovmm/configs/agent/heuristic_agent.yaml | 4 + projects/habitat_ovmm/configs/agent/rl_agent.yaml | 4 + projects/offline_mapping/build_map.py | 4 +- projects/pick_location_real_robot.py | 175 ---------------- projects/real_world_ovmm/README.md | 6 +- projects/real_world_ovmm/build_3d_map.py | 320 +++++++++++++++++++++++----- projects/real_world_ovmm/eval_episode.py | 3 +- projects/real_world_ovmm/experimental/demo_fastsam.py | 50 +++++ projects/real_world_ovmm/experimental/demo_sam.py | 71 +++++++ projects/real_world_ovmm/experimental/demo_zerodepth.py | 54 +++++ projects/real_world_ovmm/experimental/execute_trajectory.py | 125 +++++++++++ projects/real_world_ovmm/experimental/goals.txt | 3 + projects/real_world_ovmm/experimental/mapping.py | 188 +++++++++++++++++ projects/real_world_ovmm/experimental/trajectories.txt | 3 + projects/real_world_ovmm/pick_cup_real_robot.py | 28 --- src/home_robot/environment.yml | 1 + src/home_robot/home_robot/agent/objectnav_agent/objectnav_agent.py | 3 + src/home_robot/home_robot/agent/objectnav_agent/objectnav_agent_module.py | 3 + src/home_robot/home_robot/agent/ovmm_agent/__init__.py | 5 + src/home_robot/home_robot/agent/ovmm_agent/ovmm_perception.py | 14 +- src/home_robot/home_robot/agent/ovmm_agent/ppo_agent.py | 18 +- src/home_robot/home_robot/core/interfaces.py | 10 + src/home_robot/home_robot/datasets/__init__.py | 0 src/home_robot/home_robot/datasets/scannet/.gitignore | 11 + src/home_robot/home_robot/datasets/scannet/__init__.py | 6 + src/home_robot/home_robot/datasets/scannet/data/batch_load_scannet_data.py | 197 ++++++++++++++++++ src/home_robot/home_robot/datasets/scannet/data/extract_instance_images.py | 155 ++++++++++++++ src/home_robot/home_robot/datasets/scannet/data/extract_posed_images.py | 240 +++++++++++++++++++++ src/home_robot/home_robot/datasets/scannet/data/load_scannet_data.py | 226 ++++++++++++++++++++ src/home_robot/home_robot/datasets/scannet/data/scannet_utils.py | 104 +++++++++ src/home_robot/home_robot/datasets/scannet/scannet_dataset.py | 408 ++++++++++++++++++++++++++++++++++++ src/home_robot/home_robot/datasets/scannet/visualize_scannet.ipynb | 191 +++++++++++++++++ src/home_robot/home_robot/mapping/semantic/categorical_2d_semantic_map_module.py | 8 +- src/home_robot/home_robot/mapping/semantic/instance_tracking_modules.py | 372 +++++++++++++++++++++++++++------ src/home_robot/home_robot/mapping/voxel.py | 466 +++++++++++++++++++++++++++++++++++++---- src/home_robot/home_robot/mapping/voxel_map.py | 26 +++ src/home_robot/home_robot/motion/base.py | 33 ++- src/home_robot/home_robot/motion/rrt.py | 113 +++++++++- src/home_robot/home_robot/motion/space.py | 64 +++++- src/home_robot/home_robot/perception/detection/detic/detic_perception.py | 3 + src/home_robot/home_robot/perception/detection/grounded_sam/grounded_sam_perception.py | 1 + src/home_robot/home_robot/utils/__init__.py | 3 + src/home_robot/home_robot/utils/bboxes_3d.py | 1156 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/home_robot/home_robot/utils/bboxes_3d_plotly.py | 660 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/home_robot/home_robot/utils/image.py | 33 +++ src/home_robot/home_robot/utils/pose.py | 12 ++ src/home_robot/home_robot/utils/simple_env.py | 46 ++++ src/home_robot/home_robot/utils/visualization.py | 17 ++ src/home_robot_hw/home_robot_hw/env/stretch_abstract_env.py | 6 + src/home_robot_hw/home_robot_hw/env/stretch_pick_and_place_env.py | 2 +- src/home_robot_hw/home_robot_hw/remote/api.py | 41 ++++ src/home_robot_hw/home_robot_hw/remote/modules/head.py | 11 + src/home_robot_hw/home_robot_hw/remote/modules/nav.py | 49 ++++- src/home_robot_hw/home_robot_hw/remote/ros.py | 25 ++- src/home_robot_hw/home_robot_hw/ros/camera.py | 8 +- src/home_robot_hw/home_robot_hw/ros/lidar.py | 63 ++++++ src/home_robot_hw/home_robot_hw/ros/synchronized.py | 111 ++++++++++ src/home_robot_hw/launch/mapping_demo.rviz | 24 +-- src/home_robot_hw/launch/ros_workstation.rviz | 635 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/third_party/habitat-lab | 2 +- 63 files changed, 6217 insertions(+), 426 deletions(-) delete mode 100644 projects/pick_location_real_robot.py create mode 100644 projects/real_world_ovmm/experimental/demo_fastsam.py create mode 100644 projects/real_world_ovmm/experimental/demo_sam.py create mode 100644 projects/real_world_ovmm/experimental/demo_zerodepth.py create mode 100755 projects/real_world_ovmm/experimental/execute_trajectory.py create mode 100644 projects/real_world_ovmm/experimental/goals.txt create mode 100644 projects/real_world_ovmm/experimental/mapping.py create mode 100644 projects/real_world_ovmm/experimental/trajectories.txt delete mode 100644 projects/real_world_ovmm/pick_cup_real_robot.py create mode 100644 src/home_robot/home_robot/datasets/__init__.py create mode 100644 src/home_robot/home_robot/datasets/scannet/.gitignore create mode 100644 src/home_robot/home_robot/datasets/scannet/__init__.py create mode 100644 src/home_robot/home_robot/datasets/scannet/data/batch_load_scannet_data.py create mode 100644 src/home_robot/home_robot/datasets/scannet/data/extract_instance_images.py create mode 100644 src/home_robot/home_robot/datasets/scannet/data/extract_posed_images.py create mode 100644 src/home_robot/home_robot/datasets/scannet/data/load_scannet_data.py create mode 100644 src/home_robot/home_robot/datasets/scannet/data/scannet_utils.py create mode 100644 src/home_robot/home_robot/datasets/scannet/scannet_dataset.py create mode 100644 src/home_robot/home_robot/datasets/scannet/visualize_scannet.ipynb create mode 100644 src/home_robot/home_robot/mapping/voxel_map.py create mode 100644 src/home_robot/home_robot/utils/bboxes_3d.py create mode 100644 src/home_robot/home_robot/utils/bboxes_3d_plotly.py create mode 100644 src/home_robot/home_robot/utils/simple_env.py create mode 100644 src/home_robot_hw/home_robot_hw/ros/lidar.py create mode 100644 src/home_robot_hw/home_robot_hw/ros/synchronized.py create mode 100644 src/home_robot_hw/launch/ros_workstation.rviz filtering content: 100% (44/44), 370.76 MiB | 4.35 MiB/s, completed. Submodule Path： 'data/datasets/ovmm'：checkout 'c6644c74c8f4ccf4a9bd13fe4526425baa643934' error: The following untracked files in the working directory will be overwritten by the checkout operation： navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.navmesh navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.pkl Before switching branches, please move or delete. Terminating error: Your local modifications to the following files will be overwritten by the checkout operation： habitat-lab/habitat/robots/fetch_robot.py habitat-lab/habitat/robots/mobile_manipulator.py Please commit or stash your changes before switching branches. Terminating Unable to check out '2c435284686c1f598ac311c33211c8af9b6e1017' in submodule path 'data/hssd-hab'. Unable to check out '9ccca099742df667c4e4fc43e204abd8c60e6028' in submodule path 'src/third_party/habitat-lab'. ```

yvsriram commented 1 year ago

Are you still following the setup where you download the data locally and then upload it? Also, I assume you ran the above on a remote server?

In your case, errors in updating the datasets using git submodule update --init --recursive on the remote server are expected. I meant for you to first update datasets on the local computer initially.

But it seems odd that you have these files navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.navmesh, given that you downloaded and uploaded the datasets you mentioned here.

If the above log was indeed from a remote server, I suggest you to do the following:

  1. On the local computer run: a. git submodule update --init --recursive
  2. Upload datasets to remote server (you can try using rsync to make this efficient)
  3. On remote server: Did you have any uncommitted changes inside habitat-lab? If not, can you please do the following: a. cd src/third_party/habitat-lab b. git reset --hard HEAD c. git submodule update --init --recursive .
FUIGUIMURONG commented 1 year ago

I can operate on the remote server now, sorry about not making it clear with you. I do the 3. operation as you instruct. It shows the same like this:

(base) sdp@xdy-Super-Server:~/home-robot$ cd src/third_party/habitat-lab
(base) sdp@xdy-Super-Server:~/home-robot/src/third_party/habitat-lab$ git reset --hard HEAD
HEAD is now locating at 12e735d2 use unique recep names, rename navmesh folder (#1541)
(base) sdp@xdy-Super-Server:~/home-robot/src/third_party/habitat-lab$ git submodule update --init --recursive
(base) sdp@xdy-Super-Server:~/home-robot/src/third_party/habitat-lab$ cd ../..
(base) sdp@xdy-Super-Server:~/home-robot/src$ cd ..
(base) sdp@xdy-Super-Server:~/home-robot$ git submodule update --init --recursive
error: 
The following untracked files in the working directory will be overwritten by the checkout operation.：
        navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.pkl
Please move or delete before switching branches.
Terminating
Submodule path 'src/third_party/habitat-lab': Checking out '9ccca099742df667c4e4fc43e204abd8c60e6028'.
Unable to check out '2c435284686c1f598ac311c33211c8af9b6e1017' in submodule path 'data/hssd-hab'.
yvsriram commented 1 year ago

Ok great. In that case, you can just delete the two files: navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.navmesh and navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.pkl. These must have been generated when you started evaluations.

After deleting these, you can run git submodule update --init --recursive.

FUIGUIMURONG commented 1 year ago

I do delete the two files and conduct git submodule update --init --recursive. But the weird thing happens, it shows the deleted files still exist. I have checked these two files have been deleted.

  1. (base) sdp@xdy-Super-Server:~/home-robot$ git submodule update --init --recursive
error: The following untracked files in the working directory will be overwritten by the checkout operation：
    navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.navmesh
    navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.pkl
Please move or delete before switching branches.
Terminating
Unable to check out '2c435284686c1f598ac311c33211c8af9b6e1017' in submodule path 'data/hssd-hab'.
(base) sdp@xdy-Super-Server:~/home-robot$ cd data/hssd-hab/navmeshes/scenes-uncluttered/stretch_robot/
(base) sdp@xdy-Super-Server:~/home-robot/data/hssd-hab/navmeshes/scenes-uncluttered/stretch_robot$ ls
(base) sdp@xdy-Super-Server:~/home-robot/data/hssd-hab/navmeshes/scenes-uncluttered/stretch_robot$

    2.Is it right that the folder scenes-uncluttered/stretch_robot has no file?

yvsriram commented 1 year ago

These would be inside data/hssd-hab. Please add data/hssd-hab/ to start of these paths

FUIGUIMURONG commented 1 year ago

These would be inside data/hssd-hab. Please add data/hssd-hab/ to start of these paths Sorry I donnot catch your meaning, The data/hssd-hab/ is actually the start like https://github.com/facebookresearch/home-robot/issues/347#issuecomment-1697377636 shows . Would you please give me more specific instructions.

yvsriram commented 1 year ago

Yes, it looks like you have deleted these files inside data/hssd-hab. Not sure why git says that these are present. Can you share the output of cd data/hssd-hab && git status.

FUIGUIMURONG commented 1 year ago

The output shows like this:

(home-robot) sdp@xdy-Super-Server:~/home-robot/data/hssd-hab$ git status
header pointer detached at 22a8c8885
changes to commit：
  （Use "git restore --staged <file>..." to unstage）
        New file：   navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.navmesh
        New file：   navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.pkl

Changes not yet staged for commit:
   (Use "git add/rm <file>..." to update what to commit)
   (use "git restore <file>..." to discard workspace changes)
        delete：     navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.navmesh
        delete：     navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.pkl
yvsriram commented 1 year ago

Looks like these have been staged. Can you do this first git restore --staged .?

FUIGUIMURONG commented 1 year ago

After run git restore --staged ., the stage is clear. And I run git submodule update --init --recursive, but a error happens like the following. I did not do modifications locally. I am not familiar with the git and thanks for your kind instructions.

(home-robot) sdp@xdy-Super-Server:~/home-robot/data/hssd-hab$ git restore --staged .
(home-robot) sdp@xdy-Super-Server:~/home-robot/data/hssd-hab$ git status
header pointer detached at 22a8c8885
No files to commit, clean workspace
(home-robot) sdp@xdy-Super-Server:~/home-robot/data/hssd-hab$ cd ../..
(home-robot) sdp@xdy-Super-Server:~/home-robot$ git submodule update --init --recursive

^C
Exiting because of "interrupt" signal.

(home-robot) sdp@xdy-Super-Server:~/home-robot$ git submodule update --init --recursive
Error: Your local modifications to the following files will be overwritten by checkout:
        hssd-hab-uncluttered.scene_dataset_config.json
        hssd-hab.scene_dataset_config.json
        objects/0/0001fb06b075a743e6289236cf049df3ad5dfa9c.object_config.json
        objects/0/000bbe302b53fd2904e7ae92e1516a18f29de02d.object_config.json
        objects/0/0022fa6ee5a44330e765488919803155b1a9e88c.object_config.json
        objects/0/00258bed0c6e87a14e33c3eebffc48c898135698.object_config.json
        objects/0/002f87c8e4e2b870387e095797a2f40b30e7baec.object_config.json
        objects/0/0030aabddff541fbd378e38ae2d8597aecbfc757.object_config.json
        objects/0/0034b3becda06d5dbecdf52f1842e5069e126af2.glb
        objects/0/0034b3becda06d5dbecdf52f1842e5069e126af2.object_config.json
        objects/0/00366b86401aa16b702c21de49fd59b75ab9c57b.object_config.json
        objects/0/00386b5179df69aec660a71fb9c65b344857d016.object_config.json
        objects/0/003949689122adfc2370d64c0615edc1b257db83.object_config.json
        objects/0/0039699ff96422d70b73397371664863025d75e8.object_config.json
        objects/0/0040287f3af79d7b02985600e23469d319007792.object_config.json
        objects/0/0042224f50400c921142bc9777c28823dfc728ab.object_config.json
        objects/0/004315b97305e4c6986491dfc1057ae72b91537e.object_config.json
        objects/0/00474f62f75a5047b51a4e6f7072a51d0d00afef.object_config.json
        objects/0/004db3bd065eb72d8303f33c5bad163326cf1bd9.object_config.json
        objects/0/004e36a469b1e97720f54b114d251782cd41bfaa.object_config.json
        objects/0/005b673d5ded782d54be17969159917dfc9ae0f2.object_config.json
        objects/0/00662bccdee49ca148d0c96929286db6d6d3c267.glb
        objects/0/00662bccdee49ca148d0c96929286db6d6d3c267.object_config.json
        objects/0/006ea4bce9bbe0c73d6d47776b25e76265235722.object_config.json
        objects/0/0070649630e7b2262ec0427c0e2035232240fb4d.object_config.json
        objects/0/0071f864b2cc52a7ecd4603f5f92bcdcdc93a2d1.object_config.json
        objects/0/0074b6bf5758fd7186157fd3a8d53afe3e200cba.object_config.json
        objects/0/00759595cbb0f07062154e6a0b07de033871c0e1.glb
        objects/0/00759595cbb0f07062154e6a0b07de033871c0e1.object_config.json
        objects/0/00770bbb6345e851e425d78c333c4baababf3729.object_config.json
        objects/0/007ac3dc7a89e1e403c1340a4fd61fdc75ca053d.object_config.json
        objects/0/0082238889bd1b4ab20d0f9e3a444f5b61664e3a.glb
        objects/0/0082238889bd1b4ab20d0f9e3a444f5b61664e3a.object_config.json
        objects/0/00826491ee9361be3f199d05588c5dfae493f0cd.glb
        objects/0/00826491ee9361be3f199d05588c5dfae493f0cd.object_config.json
        objects/0/008cc6178827f98658c83a3b33124a0ca7ad3c26.glb
        objects/0/008cc6178827f98658c83a3b33124a0ca7ad3c26.object_config.json
        objects/0/0091ed73925f43dda0dd03d8a71b88ccfa65fbb1.object_config.json
        objects/0/0092599e2477209980084af7df385284843a3ff3.object_config.json
        objects/0/0097dd7add1d9d8fdc58a446796dcde8c6f80238.object_config.json
        objects/0/009de394cd2d691583965e0235ff47ea2d8ebf5f.glb
        objects/0/009de394cd2d691583965e0235ff47ea2d8ebf5f.object_config.json
        objects/0/009f96f4bd369a2294a1b29538c52b14ec792a54.object_config.json
        objects/0/009f9df21592e289bc08c4562f21dce15fced6fb.object_config.json
        objects/0/00a1a59b75fae9b7e42a128ce2249158dfa7d847.object_config.json
        objects/0/00a2b0f3886ccb5ffddac704f8eeec324a5e14c6.object_config.json
        objects/0/00a843989a32df31a10bc8129fa6801d768855c5.object_config.json
        objects/0/00a896b002a685b0366798b1d3a96a20e96cf560.object_config.json
        objects/0/00ab1b2fd45dcfbfdffe36f00fe21a9dddd9f06b.object_config.json
        objects/0/00b0d5e167ae6b42666de010025efad4506563f1.object_config.json
        objects/0/00b285ed70673826ccb6941929a64abfaf5f9239.object_config.json
        objects/0/00b3040d6d75d68ba6d1bdf96e31c6b8432cbc1f.object_config.json
        objects/0/00b543e5c1ec525f02523484c372d31fbcc696ea.object_config.json
        objects/0/00b94ee7f2a013ffb7b681d19a5769b63c1b8357.object_config.json
        objects/0/00ba4b1c24ae030dc052d6d196d0461db86710aa.object_config.json
        objects/0/00bcbedb6c1552d671d69e1cf51def0fdffebe49.glb
        objects/0/00bcbedb6c1552d671d69e1cf51def0fdffebe49.object_config.json
        objects/0/00bde034d3d5b08ef952aeba2bd40839d3b8c5e0.object_config.json
        objects/0/00c36e162937f13271f982b5dcafb4d3038e906a.object_config.json
        objects/0/00c
Error: The following untracked files in the workspace will be overwritten by checkout:
        navmeshes/scenes-uncluttered/stretch_robot/102344022.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102344049.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102344049.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102344193.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102344193.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102344250.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102344250.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102344280.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102344280.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102344403.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102344403.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102344457.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102344457.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102815859.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102815859.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102816009.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102816216.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102816216.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102816756.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102816756.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102817140.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102817140.pkl
        navmeshes/scenes-uncluttered/stretch_robot/102817200.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/102817200.pkl
        navmeshes/scenes-uncluttered/stretch_robot/103997424_171030444.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/103997424_171030444.pkl
        navmeshes/scenes-uncluttered/stretch_robot/103997460_171030507.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/103997586_171030669.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/103997586_171030669.pkl
        navmeshes/scenes-uncluttered/stretch_robot/103997643_171030747.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/103997643_171030747.pkl
        navmeshes/scenes-uncluttered/stretch_robot/103997895_171031182.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/103997895_171031182.pkl
        navmeshes/scenes-uncluttered/stretch_robot/103997919_171031233.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/103997919_171031233.pkl
        navmeshes/scenes-uncluttered/stretch_robot/104348082_171512994.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104348082_171512994.pkl
        navmeshes/scenes-uncluttered/stretch_robot/104348328_171513363.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104348361_171513414.pkl
        navmeshes/scenes-uncluttered/stretch_robot/104348463_171513588.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104348463_171513588.pkl
        navmeshes/scenes-uncluttered/stretch_robot/104862621_172226772.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104862621_172226772.pkl
        navmeshes/scenes-uncluttered/stretch_robot/104862639_172226823.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104862639_172226823.pkl
        navmeshes/scenes-uncluttered/stretch_robot/104862660_172226844.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104862660_172226844.pkl
        navmeshes/scenes-uncluttered/stretch_robot/104862669_172226853.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104862669_172226853.pkl
        navmeshes/scenes-uncluttered/stretch_robot/104862681_172226874.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/104862681_172226874.pkl
        navmeshes/scenes-uncluttered/stretch_robot/105515211_173104179.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/105515211_173104179.pkl
        navmeshes/scenes-uncluttered/stretch_robot/105515211_173104185.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/105515211_173104185.pkl
        navmeshes/scenes-uncluttered/stretch_robot/105515448_173104512.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/105515448_173104512.pkl
        navmeshes/scenes-uncluttered/stretch_robot/106366173_174226431.navmesh
        navmeshes/scenes-uncluttered/stretch_robot/106366173_174226431.pkl
        navmeshes/scenes-uncluttered/stretch_robo
Terminating
Fatal error: Failed to check out '2c435284686c1f598ac311c33211c8af9b6e1017' in submodule path 'data/hssd-hab'
yvsriram commented 1 year ago

I think this may be because you interrupted and ran it the second time. Can you clear navmeshes/scenes-uncluttered/stretch_robot directory again, ensure no files are staged, and then run this without interrupting?

FUIGUIMURONG commented 1 year ago

I have tried, but the same problem happens. I decide to delete the hssd-hab and re-download it to see if it's ok

FUIGUIMURONG commented 1 year ago

Thanks for your kind answers and I can run the demo. But I meet another problem：The code cannot from numba.utils import IS_PY3. It is weird because my environment has installed llvmlite-0.40.1 numba-0.57.1. I wonder if it's the problem with the high version of numba. I want to ask what is your version of numba that does not report the cannot from numba.utils import IS_PY3 problem.

cpaxton commented 1 year ago

Will close this and continue discussion in the other issue