Closed jlidard closed 8 months ago
jimmytyyang
commented
8 months ago Hi @jlidard! I hope you are doing well! Yes, you are correct, currently in the code we teleport the robot's base based on linear and angular velocities output from the social nav policy, and the legs are not animated. We will add animation code for legs later. In addition, the warning is not related to the leg animation. It is expected, so you do not need to worry about this.
jimmytyyang
commented
8 months ago Thank you for reporting this! :)
jlidard
commented
8 months ago Gotcha, thanks for clarifying @jimmytyyang!
ruddradev
commented
6 months ago For anyone looking for it, I found the working code present in the below merge request and was able to run the leg animation successfully -
Habitat-Lab and Habitat-Sim versions
Habitat-Lab: 0.3.0
Habitat-Sim: 0.3.0
Docs and Tutorials
Did you read the docs? Yes
Did you check out the tutorials? Yes
❓ Questions and Help
Hi Habitat team, many thanks for the great release and for providing example code for evaluation. My questions relate to the social navigation task as described on the habitat-baselines readme. I am able to load the checkpoint provided, and the model works reasonably well. However, using the provided evaluation code, the Spot agent doesn't appear to be articulated. In the outputs below, I noticed a warning related to incompatible light setup, also addressed in #1704. Is the warning related to this issue? Could you provide any next steps to debug?
An example video output is attached (compressed due to github file size limits):
https://github.com/facebookresearch/habitat-lab/assets/60638575/493dca81-f409-4ccf-bda6-66bdff9cae94
Run params:
--config-name=social_nav/social_nav.yaml benchmark/multi_agent=hssd_spot_human_social_nav habitat_baselines.evaluate=True habitat_baselines.num_checkpoints=3 habitat_baselines.total_num_steps=1.0e9 habitat_baselines.num_environments=1 habitat_baselines.video_dir=video_social_nav habitat_baselines.checkpoint_folder=checkpoints_social_nav habitat_baselines.eval_ckpt_path_dir=checkpoints_social_nav/social_nav_latest.pth habitat.task.actions.agent_0_base_velocity.longitudinal_lin_speed=10.0 habitat.task.actions.agent_0_base_velocity.ang_speed=10.0 habitat.task.actions.agent_0_base_velocity.allow_dyn_slide=True habitat.task.actions.agent_0_base_velocity.enable_rotation_check_for_dyn_slide=False habitat.task.actions.agent_1_oracle_nav_randcoord_action.human_stop_and_walk_to_robot_distance_threshold=-1.0 habitat.task.actions.agent_1_oracle_nav_randcoord_action.lin_speed=10.0 habitat.task.actions.agent_1_oracle_nav_randcoord_action.ang_speed=10.0 habitat.task.actions.agent_1_oracle_nav_action.lin_speed=10.0 habitat.task.actions.agent_1_oracle_nav_action.ang_speed=10.0 habitat.task.measurements.social_nav_reward.facing_human_reward=3.0 habitat.task.measurements.social_nav_reward.count_coll_pen=0.01 habitat.task.measurements.social_nav_reward.max_count_colls=-1 habitat.task.measurements.social_nav_reward.count_coll_end_pen=5 habitat.task.measurements.social_nav_reward.use_geo_distance=True habitat.task.measurements.social_nav_reward.facing_human_dis=3.0 habitat.task.measurements.social_nav_seek_success.following_step_succ_threshold=400 habitat.task.measurements.social_nav_seek_success.need_to_face_human=True habitat.task.measurements.social_nav_seek_success.use_geo_distance=True habitat.task.measurements.social_nav_seek_success.facing_threshold=0.5 habitat.task.lab_sensors.humanoid_detector_sensor.return_image=True habitat.task.lab_sensors.humanoid_detector_sensor.is_return_image_bbox=True habitat.task.success_reward=10.0 habitat.task.end_on_success=False habitat.task.slack_reward=-0.1 habitat.environment.max_episode_steps=1500 habitat.simulator.kinematic_mode=True habitat.simulator.ac_freq_ratio=4 habitat.simulator.ctrl_freq=120 habitat.simulator.agents.agent_0.joint_start_noise=0.0 habitat_baselines.load_resume_state_config=False habitat_baselines.test_episode_count=5 habitat_baselines.eval.extra_sim_sensors.third_rgb_sensor.height=1080 habitat_baselines.eval.extra_sim_sensors.third_rgb_sensor.width=1920Outputs:
Renderer: NVIDIA GeForce RTX 4090/PCIe/SSE2 by NVIDIA Corporation OpenGL version: 4.6.0 NVIDIA 545.29.02 Using optional features: GL_ARB_vertex_array_object GL_ARB_separate_shader_objects GL_ARB_robustness GL_ARB_texture_storage GL_ARB_texture_view GL_ARB_framebuffer_no_attachments GL_ARB_invalidate_subdata GL_ARB_texture_storage_multisample GL_ARB_multi_bind GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_texture_filter_anisotropic GL_KHR_debug GL_KHR_parallel_shader_compile GL_NV_depth_buffer_float Using driver workarounds: no-forward-compatible-core-context nv-egl-incorrect-gl11-function-pointers no-layout-qualifiers-on-old-glsl nv-zero-context-profile-mask nv-implementation-color-read-format-dsa-broken nv-cubemap-inconsistent-compressed-image-size nv-cubemap-broken-full-compressed-image-query nv-compressed-block-size-in-bits [16:31:33:545698]:[Warning]:[Metadata] SceneDatasetAttributes.cpp(105)::addNewSceneInstanceToDataset : Dataset : 'hssd-hab-uncluttered' : Lighting Layout Attributes 'data/scene_datasets/hssd-hab/scenes-uncluttered/107734176_176000019.scene_instance.json' specified in Scene Attributes but does not exist in dataset, so creating default. [16:31:33:600324]:[Warning]:[Sim] Simulator.cpp(533)::instanceStageForSceneAttributes : The active scene does not contain semantic annotations : activeSemanticSceneID_ = 0 2023-12-18 16:31:34,540 Initializing task RearrangePddlSocialNavTask-v0 /home/jlidard/miniconda3/envs/habitat/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") /home/jlidard/miniconda3/envs/habitat/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") 2023-12-18 16:31:35,270 Number of params to train: 8412805 2023-12-18 16:31:35,271 Agent number of parameters: 8412805 2023-12-18 16:31:35,295 Number of params to train: 0 2023-12-18 16:31:35,295 Agent number of parameters: 0 [16:31:35:379687]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/base.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380434]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/fl.hip.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380460]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/fl.uleg.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380486]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/fl.lleg.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380502]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/fr.hip.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380518]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/fr.uleg.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380542]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/fr.lleg.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380557]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/hl.hip.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380572]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/hl.uleg.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380596]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/hl.lleg.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380611]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/hr.hip.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380625]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/hr.uleg.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380648]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/hr.lleg.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380663]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/arm0.link_sh0.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380687]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/arm0.link_sh1.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380700]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/arm0.link_hr0.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380769]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/arm0.link_el0.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380822]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/arm0.link_el1.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380885]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/arm0.link_wr0.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380906]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/arm0.link_wr1.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. [16:31:35:380940]:[Warning]:[Assets] ResourceManager.cpp(945)::createRenderAssetInstance : Instantiating render assetdata/robots/hab_spot_arm/urdf/../meshesColored/arm0.link_fngr.glbwith incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. MeshTools::compile(): ignoring unknown/unsupported attribute Trade::MeshAttribute::Custom(0) MeshTools::compile(): ignoring unknown/unsupported attribute Trade::MeshAttribute::Custom(1) /home/jlidard/habitat-lab/habitat-lab/habitat/tasks/rearrange/robot_specific_sensors.py:67: UserWarning: You are using SpotHeadStereoDepthSensor but you do not provide head_stereo_right_depth and head_stereo_right_depth.We return an all zero image.