[Social Nav] Error during evaluation ('NoneType' object is not subscriptable.)

[qwerty863]

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: vx.x.x or master? master

Habitat-Sim: vx.x.x or master? master

Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release versions of Habitat-Lab and Habitat-Sim? Your question may already be addressed in the latest versions. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.

Master branch contains 'bleeding edge' code and should be used at your own risk.

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-lab/ Yes

Did you check out the tutorials? https://aihabitat.org/tutorial/2020/ Yes

Perhaps your question is answered there. If not, carry on!

❓ Questions and Help

Hi, I'm evaluating my reproduced social navigation agent following the command given by @jimmytyyang with a different habitat_baselines.test_episode_count value (500->1200) since the validation scenes are 12 and each has 100 episodes. https://github.com/facebookresearch/habitat-lab/tree/main/habitat-baselines

But I got this error ('NoneType' object is not subscriptable). Has this error been seen before? Or am I missing something?

 78%|██████████████████▉               | 939/1200 [3:21:16<1:04:45, 14.89s/it][ERROR,rearrange_task] 2023-12-17 06:47:18,091 pddl_sim_state.py:217 Failed to place the robot.
 79%|███████████████████                | 943/1200 [3:22:15<58:38, 13.69s/it]Error executing job with overrides: ['benchmark/multi_agent=hssd_spot_human_social_nav', 'habitat_baselines.evaluate=True', 'habitat_baselines.num_checkpoints=5000', 'habitat_baselines.total_num_steps=1.0e9', 'habitat_baselines.num_environments=12', 'habitat_baselines.checkpoint_folder=checkpoints_social_nav/reproduce2', 'habitat_baselines.eval_ckpt_path_dir=checkpoints_social_nav/reproduce2/latest.pth', 'habitat.task.actions.agent_0_base_velocity.longitudinal_lin_speed=10.0', 'habitat.task.actions.agent_0_base_velocity.ang_speed=10.0', 'habitat.task.actions.agent_0_base_velocity.allow_dyn_slide=True', 'habitat.task.actions.agent_0_base_velocity.enable_rotation_check_for_dyn_slide=False', 'habitat.task.actions.agent_1_oracle_nav_randcoord_action.human_stop_and_walk_to_robot_distance_threshold=-1.0', 'habitat.task.actions.agent_1_oracle_nav_randcoord_action.lin_speed=10.0', 'habitat.task.actions.agent_1_oracle_nav_randcoord_action.ang_speed=10.0', 'habitat.task.actions.agent_1_oracle_nav_action.lin_speed=10.0', 'habitat.task.actions.agent_1_oracle_nav_action.ang_speed=10.0', 'habitat.task.measurements.social_nav_reward.facing_human_reward=3.0', 'habitat.task.measurements.social_nav_reward.count_coll_pen=0.01', 'habitat.task.measurements.social_nav_reward.max_count_colls=-1', 'habitat.task.measurements.social_nav_reward.count_coll_end_pen=5', 'habitat.task.measurements.social_nav_reward.use_geo_distance=True', 'habitat.task.measurements.social_nav_reward.facing_human_dis=3.0', 'habitat.task.measurements.social_nav_seek_success.following_step_succ_threshold=400', 'habitat.task.measurements.social_nav_seek_success.need_to_face_human=True', 'habitat.task.measurements.social_nav_seek_success.use_geo_distance=True', 'habitat.task.measurements.social_nav_seek_success.facing_threshold=0.5', 'habitat.task.lab_sensors.humanoid_detector_sensor.return_image=True', 'habitat.task.lab_sensors.humanoid_detector_sensor.is_return_image_bbox=True', 'habitat.task.success_reward=10.0', 'habitat.task.end_on_success=False', 'habitat.task.slack_reward=-0.1', 'habitat.environment.max_episode_steps=1500', 'habitat.simulator.kinematic_mode=True', 'habitat.simulator.ac_freq_ratio=4', 'habitat.simulator.ctrl_freq=120', 'habitat.simulator.agents.agent_0.joint_start_noise=0.0', 'habitat_baselines.load_resume_state_config=False', 'habitat_baselines.test_episode_count=1200']
Traceback (most recent call last):
  File "/home/user/habitat-lab/habitat-baselines/habitat_baselines/run.py", line 31, in main
    execute_exp(cfg, "eval" if cfg.habitat_baselines.evaluate else "train")
  File "/home/user/habitat-lab/habitat-baselines/habitat_baselines/run.py", line 62, in execute_exp
    trainer.eval()
  File "/home/user/habitat-lab/habitat-baselines/habitat_baselines/common/base_trainer.py", line 129, in eval
    self._eval_checkpoint(
  File "/home/user/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/ppo_trainer.py", line 889, in _eval_checkpoint
    evaluator.evaluate_agent(
  File "/home/user/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/habitat_evaluator.py", line 294, in evaluate_agent
    ) = pause_envs(
  File "/home/user/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/evaluator.py", line 89, in pause_envs
    rgb_frames = [rgb_frames[i] for i in state_index]
  File "/home/user/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/evaluator.py", line 89, in <listcomp>
    rgb_frames = [rgb_frames[i] for i in state_index]
TypeError: 'NoneType' object is not subscriptable

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Thank you in advance!

[yubinwang11]

Same issues with me.

[jimmytyyang]

Hi @yubinwang11 and @qwerty863, I hope you are doing well! Thank you for reporting this. Yes, this is a known issue when we evaluate the model in the multi-agent setting with test_episode_count value close to the total number of episodes. The reason is that some workers finish the job first, and they do not have any episode to evaluate, causing a "None" issue. We will fix that asap.

[jimmytyyang]

Hi @yubinwang11 and @qwerty863, are you able to use 500 and make it runnable?

[qwerty863]

Hi @jimmytyyang, thanks for replying! Yes, I was able to run 500 with the command that you provided.