Failed to Run Demo Setup

xiaotaw commented 10 months ago

🐛 Bug

Steps to Reproduce

clone home-robot and update all submodules


xt@ubt:/mnt/xt/8T/CODES/EmbodiedAI/home-robot$ git submodule 
9ad25fbd86a3fd352c7a0fc1f99132fbb5802378 data/datasets/ovmm (heads/main)
4e0292b16b32c92267be1ac8a94628deccd2ef09 data/hssd-hab (v0.2.3-8-g4e0292b16)
3893a735352b92d46505f35d759553f5fc82a39b data/objects (heads/main)
ddcf11f42e7e0732a0c4607648f9448ea8d73590 src/home_robot/home_robot/agent/imagenav_agent/SuperGluePretrainedNetwork (heads/master)
46f8317a0688b4b94e85816b5839a1dbca26d1e0 src/home_robot/home_robot/perception/detection/detic/Detic (46f8317)
6bb6a22068c79bb36a733d25414c21bd8b650416 src/home_robot/home_robot/perception/detection/grounded_sam/Grounded-Segment-Anything (6bb6a22)
f26b8e7196c22a8c12fc0d99b1876a274a345f91 src/third_party/contact_graspnet (remotes/origin/cpaxton/home-robot)
38af375052d3ae7331141bc1a22cfa2713b02987 src/third_party/detectron2 (v0.6-152-g38af375)
9ccca099742df667c4e4fc43e204abd8c60e6028 src/third_party/habitat-lab (v0.1.4-449-g9ccca099)
xt@ubt:/mnt/xt/8T/CODES/EmbodiedAI/home-robot$ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files: (use "git add ..." to include in what will be committed)

bin/
run.md

nothing added to commit but untracked files present (use "git add" to track) xt@ubt:/mnt/xt/8T/CODES/EmbodiedAI/home-robot$

- run a container using  fairembodied/habitat-challenge:homerobot-ovmm-challenge-2023, with home-robot mapped into the container
-  setup in container, without a real robot
    - install mamba
    - create env, and activate
    ```bash
    mamba env create -n home-robot -f src/home_robot/environment.yml
    mamba activate home-robot

-  install home-robot and third-party
````bash
# Install the core home_robot package
python -m pip install -e src/home_robot

# install detectron2
python -m pip install -e src/third_party/detectron2

# Install Detic
cd $HOME_ROBOT_ROOT/src/home_robot/home_robot/perception/detection/detic/Detic/
mkdir models
wget https://dl.fbaipublicfiles.com/detic/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth -O models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth --no-check-certificate
cd -

# Download the Robot Model
mkdir -p data/robots/hab_stretch
cd data/robots/hab_stretch

wget http://dl.fbaipublicfiles.com/habitat/robots/hab_stretch_v1.0.zip
unzip hab_stretch_v1.0.zip
cd -

# Download pretrained skills
mkdir -p data/checkpoints
cd data/checkpoints
wget https://dl.fbaipublicfiles.com/habitat/data/baselines/v1/ovmm_baseline_home_robot_challenge_2023_v0.2.zip
unzip ovmm_baseline_home_robot_challenge_2023_v0.2.zip -d ovmm_baseline_home_robot_challenge_2023_v0.2
cd ../../
````

Run Demo Setup, and it failed

(home-robot) root@e0b90c917e7b:/mnt/xt/8T/CODES/EmbodiedAI/home-robot# python projects/habitat_ovmm/eval_baselines_agent.py --env_config projects/habitat_ovmm/configs/env/hssd_demo.yaml
+ python projects/habitat_ovmm/eval_baselines_agent.py --env_config projects/habitat_ovmm/configs/env/hssd_demo.yaml
pybullet build time: May 20 2022 19:45:31
Neither `ifconfig` (`ifconfig -a`) nor `ip` (`ip address show`) commands are available, listing network interfaces is likely to fail
An exception occurred in telemetry logging.Disabling telemetry to prevent further exceptions.
Traceback (most recent call last):
  File "/root/micromamba/envs/home-robot/lib/python3.9/site-packages/iopath/common/file_io.py", line 946, in __log_tmetry_keys
    handler.log_event()
  File "/root/micromamba/envs/home-robot/lib/python3.9/site-packages/iopath/common/event_logger.py", line 97, in log_event
    del self._evt
AttributeError: _evt
Loading pretrained CLIP
/root/micromamba/envs/home-robot/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1694449361437/work/aten/src/ATen/native/TensorShape.cpp:3483.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
2023-10-09 02:48:45,911 Initializing dataset OVMMDataset-v0
2023-10-09 02:48:46,516 initializing sim OVMMSim-v0
[02:48:48:932556]:[Core] ManagedContainerBase.h(329)::checkExistsWithMessage : <Lighting Layout>::getObjectByHandle : Unknown Lighting Layout managed object handle :  . Aborting
2023-10-09 02:48:50,162 Initializing task OVMMNavToObjTask-v0
Traceback (most recent call last):
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/projects/habitat_ovmm/eval_baselines_agent.py", line 99, in <module>
    metrics = evaluator.evaluate(
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/projects/habitat_ovmm/evaluator.py", line 458, in evaluate
    self._env = create_ovmm_env_fn(self.config)
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/projects/habitat_ovmm/utils/env_utils.py", line 34, in create_ovmm_env_fn
    habitat_env = env_class(config=habitat_config, dataset=dataset)
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 117, in __init__
    base_env = RLTaskEnv(config=config, dataset=dataset)
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 43, in __init__
    super().__init__(config, dataset)
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 382, in __init__
    self._env = Env(config, dataset)
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 118, in __init__
    self._task = make_task(
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/registration.py", line 23, in make_task
    return _task(**kwargs)
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/ovmm/sub_tasks/nav_to_obj_task.py", line 21, in __init__
    super().__init__(
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/sub_tasks/nav_to_obj_task.py", line 44, in __init__
    super().__init__(
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/rearrange_task.py", line 67, in __init__
    super().__init__(sim=sim, dataset=dataset, **kwargs)
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/nav/nav.py", line 1412, in __init__
    super().__init__(config=config, sim=sim, dataset=dataset)
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/embodied_task.py", line 249, in __init__
    self._init_entities(
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/embodied_task.py", line 274, in _init_entities
    entities[entity_name] = entity_type(
  File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/ovmm/ovmm_sensors.py", line 101, in __init__
    self._embeddings = pickle.load(f)
_pickle.UnpicklingError: invalid load key, 'v'.

Expected behavior

Run demo in a container.

Additional context

The first problem is about pillow, and I solved it via pip install pillow==7.1.1

Traceback (most recent call last):
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/projects/habitat_ovmm/eval_baselines_agent.py", line 18, in <module>
from home_robot.agent.ovmm_agent.ovmm_agent import OpenVocabManipAgent
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/home_robot/home_robot/agent/ovmm_agent/__init__.py", line 7, in <module>
from .ovmm_agent import OpenVocabManipAgent, Skill, get_skill_as_one_hot_dict
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/home_robot/home_robot/agent/ovmm_agent/ovmm_agent.py", line 15, in <module>
from home_robot.agent.ovmm_agent.ovmm_perception import (
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/home_robot/home_robot/agent/ovmm_agent/ovmm_perception.py", line 12, in <module>
from home_robot.perception.detection.detic.detic_perception import DeticPerception
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/home_robot/home_robot/perception/detection/detic/detic_perception.py", line 17, in <module>
from detectron2.data import MetadataCatalog
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/detectron2/detectron2/data/__init__.py", line 2, in <module>
from . import transforms  # isort:skip
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/detectron2/detectron2/data/transforms/__init__.py", line 4, in <module>
from .transform import *
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/detectron2/detectron2/data/transforms/transform.py", line 36, in <module>
class ExtentTransform(Transform):
File "/mnt/xt/8T/CODES/EmbodiedAI/home-robot/src/third_party/detectron2/detectron2/data/transforms/transform.py", line 46, in ExtentTransform
def __init__(self, src_rect, output_size, interp=Image.LINEAR, fill=0):
AttributeError: module 'PIL.Image' has no attribute 'LINEAR'

yvsriram commented 10 months ago

Hi,

Can you please try:cd data/objects and git lfs pull. It is likely that the data/objects/clip_embeddings.pickle is not downloaded correctly.
If you are wanting to participate in the challenge due in the next two days, please use the tag home-robot-ovmm-challenge-2023-v0.1.2 using: git clone https://github.com/facebookresearch/home-robot --branch home-robot-ovmm-challenge-2023-v0.1.2 as indicated in the README. From the logs you shared, it seems like you are using the latest main.

xiaotaw commented 10 months ago

thx

facebookresearch / home-robot

Failed to Run Demo Setup #420

🐛 Bug

Steps to Reproduce

Expected behavior

Additional context