"IndexError: list index out of range" when running objnav_benchmark.py

[Elycyx]

Hello! When I tried to evaluate the ObjNav Benchmark, I found the following problems: When I run

python objnav_benchmark.py --checkpoint="./checkpoints/pixelnav_A.ckpt"

Some errors emerged.

pybullet build time: Oct 21 2024 13:55:32
PluginManager::Manager: duplicate static plugin StbImageImporter, ignoring
PluginManager::Manager: duplicate static plugin GltfImporter, ignoring
PluginManager::Manager: duplicate static plugin BasisImporter, ignoring
PluginManager::Manager: duplicate static plugin AssimpImporter, ignoring
PluginManager::Manager: duplicate static plugin AnySceneImporter, ignoring
PluginManager::Manager: duplicate static plugin AnyImageImporter, ignoring
FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
2024-11-13 22:55:25,620 Initializing dataset ObjectNav-v1
2024-11-13 22:55:27,497 initializing sim Sim-v0
PluginManager::Manager: duplicate static plugin StbImageImporter, ignoring
PluginManager::Manager: duplicate static plugin GltfImporter, ignoring
PluginManager::Manager: duplicate static plugin BasisImporter, ignoring
PluginManager::Manager: duplicate static plugin AssimpImporter, ignoring
PluginManager::Manager: duplicate static plugin AnySceneImporter, ignoring
PluginManager::Manager: duplicate static plugin AnyImageImporter, ignoring
[22:55:27:712776]:[Error]:[Scene] SemanticScene.cpp(137)::loadSemanticSceneDescriptor : SSD Load Failure! File with SemanticAttributes-provided name `/mnt/sda1/habitat-lab/data/scene_datasets/hm3d_v0.2/val/00813-svBbv1Pavdk/svBbv1Pavdk.basis.scn` exists but failed to load.
2024-11-13 22:55:28,204 Initializing task ObjectNav-v1
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3609.)
final text_encoder_type: bert-base-uncased
FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  0%|                                                                                                                                                                            | 0/200 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/sda1/Pixel-Navigator/objnav_benchmark.py", line 61, in <module>
    obs = habitat_env.reset()
  File "/mnt/sda1/habitat-lab/habitat-lab/habitat/core/env.py", line 264, in reset
    self._task.measurements.reset_measures(
  File "/mnt/sda1/habitat-lab/habitat-lab/habitat/core/embodied_task.py", line 166, in reset_measures
    measure.reset_metric(*args, **kwargs)
  File "/mnt/sda1/habitat-lab/habitat-lab/habitat/tasks/nav/nav.py", line 857, in reset_metric
    self._draw_goals_aabb(episode)
  File "/mnt/sda1/habitat-lab/habitat-lab/habitat/tasks/nav/nav.py", line 770, in _draw_goals_aabb
    sem_scene.objects[object_id].id.split("_")[-1]
IndexError: list index out of range

I have already executed the evaluate_policy.py successfully, so I believe most of my environment setup is correct. But it is noticed that when running the evaluate_policy.py, some errors like

[22:32:37:417723]:[Error]:[Scene] SemanticScene.cpp(137)::loadSemanticSceneDescriptor : SSD Load Failure! File with SemanticAttributes-provided name /mnt/sda1/habitat-lab/data/scene_datasets/hm3d_v0.2/val/00814-p53SfW6mjZe/p53SfW6mjZe.basis.scn exists but failed to load.

happened sometimes, though they did not have obvious impacts on the results.

Does anyone know how to solve it?

[wzcai99]

@Elycyx Maybe you can set the https://github.com/wzcai99/Pixel-Navigator/blob/f7077a4cfc3fc0d320f73beafcda75375dc1ddb0/config_utils.py#L32 as False and try to run the code again.

[Elycyx]

@Elycyx Maybe you can set the

https://github.com/wzcai99/Pixel-Navigator/blob/f7077a4cfc3fc0d320f73beafcda75375dc1ddb0/config_utils.py#L32

as False and try to run the code again.

Thank you for your reply! When I set draw_goal_aabbs=False, the errors above disappear, but I got a new error:

Traceback (most recent call last):
  File "/mnt/sda1/Pixel-Navigator/objnav_benchmark.py", line 77, in <module>
    goal_image,goal_mask,debug_image,goal_rotate,goal_flag = nav_planner.make_plan(episode_images[-12:])
  File "/mnt/sda1/Pixel-Navigator/gpt4v_planner.py", line 44, in make_plan
    direction,goal_flag = self.query_gpt4v(pano_images)
  File "/mnt/sda1/Pixel-Navigator/gpt4v_planner.py", line 92, in query_gpt4v
    self.gptv_trajectory.append("GPT-4V Answer:\n%s"%raw_answer)
UnboundLocalError: local variable 'raw_answer' referenced before assignment

It seems that the agent cannot successfully infer the answer. If I add raw_answer = None in line 81, the file can be run. But due to the lack of the right answer from inference, the agent cannot move in the right direction. And the log is like this:

  0%|▊                                                                                                                                                              | 1/200 [04:08<13:42:57, 248.13s/it]No semantic annotations found
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1189, 1024) to (1200, 1024) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[rawvideo @ 0x5cdbbcf034c0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[swscaler @ 0x5cdbbcf1cfc0] Warning: data is not aligned! This can lead to a speed loss

[wzcai99]

It means that the gpt4v doesn't work. Did you prepare your openai_key in the environment variables? You should check the file https://github.com/wzcai99/Pixel-Navigator/blob/master/llm_utils/gpt_request.py and try a unit test about the API calling functions.

[Elycyx]

It means that the gpt4v doesn't work. Did you prepare your openai_key in the environment variables? You should check the file https://github.com/wzcai99/Pixel-Navigator/blob/master/llm_utils/gpt_request.py and try a unit test about the API calling functions.

Thanks a lot! I find that the code uses AzureOpenAI API while I used the standard OpenAI API. After changing the way of calling API, the code can be run, and the agent can move towards the right direction.

However, I still face the problem:

[rawvideo @ 0x5df61bd784c0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[swscaler @ 0x5df61bd91fc0] Warning: data is not aligned! This can lead to a speed loss

[wzcai99]

Those are some warnings from writing data into mp4 using imageio. But it doesn't matter.

[Elycyx]

Those are some warnings from writing data into mp4 using imageio. But it doesn't matter.

OK, that makes sense. Thank you for your time in helping me solve these problems, you really did an excellent job!

[wzcai99]

Thank you for your interest!