Error in referencing yolo sensors

[AnandSingh-0619]

https://github.com/AnandSingh-0619/home-robot/blob/ddea582082e33587e9bcf902281dcaa1d0d4a938/projects/habitat_uncertainity/utils/run.py#L99

I am unable to reference the yolo_object_segmentation sensor in the code. I am getting the following error from gym_wrapper Traceback (most recent call last): File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 268, in _worker_env observations = env.reset() File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_env_episode_count_wrapper.py", line 61, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_env_obs_dict_wrapper.py", line 36, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/site-packages/gym/core.py", line 283, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 319, in reset return self._transform_obs(obs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 282, in _transform_obs [(k, obs[k]) for k in self._gym_obs_keys] File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 282, in [(k, obs[k]) for k in self._gym_obs_keys] KeyError: 'yolo_object_segmentation' Process ForkServerProcess-4: Traceback (most recent call last): File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 268, in _worker_env observations = env.reset() File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_env_episode_count_wrapper.py", line 61, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_env_obs_dict_wrapper.py", line 36, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/site-packages/gym/core.py", line 283, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 319, in reset return self._transform_obs(obs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 282, in _transform_obs [(k, obs[k]) for k in self._gym_obs_keys] File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 282, in [(k, obs[k]) for k in self._gym_obs_keys] KeyError: 'yolo_object_segmentation' Process ForkServerProcess-3: Traceback (most recent call last): File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 268, in _worker_env observations = env.reset() File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_env_episode_count_wrapper.py", line 61, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_env_obs_dict_wrapper.py", line 36, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/site-packages/gym/core.py", line 283, in reset return self.env.reset(kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 319, in reset return self._transform_obs(obs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 282, in _transform_obs [(k, obs[k]) for k in self._gym_obs_keys] File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 282, in [(k, obs[k]) for k in self._gym_obs_keys] KeyError: 'yolo_object_segmentation' Traceback (most recent call last): File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/nethome/asingh3064/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in cli.main() File "/nethome/asingh3064/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main run() File "/nethome/asingh3064/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file runpy.run_path(target, run_name="main") File "/nethome/asingh3064/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path return _run_module_code(code, init_globals, run_name, File "/nethome/asingh3064/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code _run_code(code, mod_globals, init_globals, File "/nethome/asingh3064/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code exec(code, run_globals) File "/nethome/asingh3064/flash/home-robot/projects/habitat_uncertainity/utils/run.py", line 147, in main() File "/nethome/asingh3064/flash/home-robot/projects/habitat_uncertainity/utils/run.py", line 99, in main execute_exp(config, "train") File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-baselines/habitat_baselines/run.py", line 60, in execute_exp trainer.train() File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/contextlib.py", line 79, in inner return func(args, **kwds) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/ppo_trainer.py", line 665, in train self._init_train(resume_state) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/ppo_trainer.py", line 282, in _init_train observations = self.envs.reset() File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 389, in reset results.append(read_fn()) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 110, in call res = self.read_fn() File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 68, in recv buf = self.recv_bytes() File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError Exception ignored in: <function VectorEnv.del at 0x7ff15e250700> Traceback (most recent call last): File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 615, in del File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 467, in close File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 110, in call File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 68, in recv File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/connection.py", line 216, in recv_bytes File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/connection.py", line 383, in _recv EOFError:

[yusufali98]

Can you clarify a few things:

1. What is the exact config file you are using ?
2. Have you added the custom sensors to the config file as part of habitat/task/lab_sensors ? I sent you a clarification regarding this on Slack after our offline discussion (here). Pasting here anyway: "You will have to also overwrite the /habitat/task/lab_sensors as the rl_skill.yaml derives from this (consider if you are training gaze skill) which also specifies object_segmentation as the lab_sensor. The "object_segmentation_sensor" that is defined links to the actual sensor class through the default_structured_configs (and this). As you can see in the data_class, the name of the sensor class is used which is exactly the name of the sensor class and the linking is basically being done through the habitat global registry as we register the sensor class as a "sensor" I think the most important thing to overwrite are the /habitat/task/lab_sensors so make sure you do that. Just overwriting the obs_transforms is not enough (here, here and here)"
3. What is the exact command you use to run the code right now ?
4. Can you set the environment variable in the terminal you are running the code in: export HABITAT_ENV_DEBUG=1. This will help to debug things better with more informative error messages.

[AnandSingh-0619]

1. Config file

2. Yes
3. Debug the file run.py in habitat_uncertainity/utils

Error: Process ForkServerProcess-1: Traceback (most recent call last): File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 253, in _worker_env env = EnvCountEpisodeWrapper(EnvObsDictWrapper(env_fn(env_fn_args))) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 100, in call return self.fn(args, kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_definitions.py", line 61, in make_gym_from_config return make_env_fn(env_class=env_class, config=config, dataset=dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/utils/env_utils.py", line 37, in make_env_fn env = env_class(config=config, dataset=dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 126, in init base_env = RLTaskEnv(config=config, dataset=dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 46, in init super().init(config, dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 382, in init self._env = Env(config, dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 118, in init self._task = make_task( File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/registration.py", line 23, in make_task return _task(kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/sub_tasks/pick_task.py", line 30, in init super().init( File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/rearrange_task.py", line 75, in init super().init(sim=sim, dataset=dataset, kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/nav/nav.py", line 1413, in init super().init(config=config, sim=sim, dataset=dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/embodied_task.py", line 257, in init self._init_entities( File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/embodied_task.py", line 282, in _init_entities assert ( AssertionError: invalid yolo_object_segmentation_sensor type YOLOObjectSegmentationSensor Process ForkServerProcess-2: Traceback (most recent call last): File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/coc/pskynet4/asingh3064/miniforge3/envs/home-robot/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/vector_env.py", line 253, in _worker_env env = EnvCountEpisodeWrapper(EnvObsDictWrapper(env_fn(env_fn_args))) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 100, in call return self.fn(args, kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/gym/gym_definitions.py", line 61, in make_gym_from_config return make_env_fn(env_class=env_class, config=config, dataset=dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/utils/env_utils.py", line 37, in make_env_fn env = env_class(config=config, dataset=dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 126, in init base_env = RLTaskEnv(config=config, dataset=dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/environments.py", line 46, in init super().init(config, dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 382, in init self._env = Env(config, dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/env.py", line 118, in init self._task = make_task( File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/registration.py", line 23, in make_task return _task(kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/sub_tasks/pick_task.py", line 30, in init super().init( File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/rearrange/rearrange_task.py", line 75, in init super().init(sim=sim, dataset=dataset, kwargs) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/tasks/nav/nav.py", line 1413, in init super().init(config=config, sim=sim, dataset=dataset) File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/embodied_task.py", line 257, in init self._init_entities( File "/coc/pskynet4/asingh3064/home-robot/src/third_party/habitat-lab/habitat-lab/habitat/core/embodied_task.py", line 282, in _init_entities assert ( AssertionError: invalid yolo_object_segmentation_sensor type YOLOObjectSegmentationSensor

The error is from file . There is still some problem in sensor registration. It cannot define entity type

[yusufali98]

Few suggestions:

1. I think the issue is that you are not actually importing the custom sensor classes that you have defined in your code. Just adding this registration line is not enough. You also actually need to import the sensor classes as is done in the init.py in the ovon codebase. So for now, you can just import the sensor classes directly in your run.py.
2. Please set the export HABITAT_ENV_DEBUG=1 environment variable as it will be easier to debug if the code is run in multi-threaded fashion instead of multiprocessing scheme.
3. Why are all your files under a "utils" folder ? Please follow the ovon codebase structure exactly. Place the sensors.py in a separate tasks folder which itself is in a src folder under the habitat_uncertainty folder. The run.py will also go into the src folder. The configs folder should stay outside the src folder directly under the habitat_uncertainty folder

[yusufali98]

Also, whenever you're sending error logs from code please put them into log.txt file as it is cumbersome to read through the entire error in the comment. Share the log.txt in the comment so I can view it separately.