Question about own scene rotation in Habitat 3.0

[zoeyliu1999]

Habitat-Sim version

v0.3.1

Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release version of Habitat-Sim? Your question may already be addressed in the latest version. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.

Main branch contains 'bleeding edge' code and should be used at your own risk.

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-sim/

Did you check out the tutorials? https://aihabitat.org/tutorial/2020/

Perhaps your question is answered there. If not, carry on!

❓ Questions and Help

Hi, thank you for your excellent work! I am trying to import my own scene(bedroom_w_light.glb.zip) into Habitat 3.0, but the scene appears rotated incorrectly (please see the image below).

Based on previous issues, I suspect this might be related to the gravity axis. To address this, I tried rotating the entire scene by 90 degrees along the x-axis in Unity, but unfortunately, this did not resolve the issue.

Below is the script I used for adapting my own scene. Could you please provide some advice on how to fix this rotation problem? Thank you!

import gym
import magnum as mn
import numpy as np
import pytest
from omegaconf import DictConfig

import habitat.articulated_agents.humanoids.kinematic_humanoid as kinematic_humanoid
import habitat_sim
import habitat_sim.agent
from habitat.articulated_agent_controllers import (
    HumanoidRearrangeController,
    HumanoidSeqPoseController,
)

default_sim_settings = {
    # settings shared by example.py and benchmark.py
    "max_frames": 1000,
    "width": 640,
    "height": 480,
    "default_agent": 0,
    "sensor_height": 1.5,
    "hfov": 90,
    "color_sensor": True,  # RGB sensor (default: ON)
    "semantic_sensor": False,  # semantic sensor (default: OFF)
    "depth_sensor": False,  # depth sensor (default: OFF)
    "seed": 1,
    "silent": False,  # do not print log info (default: OFF)
    # settings exclusive to example.py
    "save_png": False,  # save the pngs to disk (default: OFF)
    "print_semantic_scene": False,
    "print_semantic_mask_stats": False,
    "compute_shortest_path": False,
    "compute_action_shortest_path": False,
    "scene": "data/scene_datasets/habitat-test-scenes/skokloster-castle.glb",
    "test_scene_data_url": None,
    # "test_scene_data_url": "http://dl.fbaipublicfiles.com/habitat/habitat-test-scenes.zip",
    "goal_position": [5.047, 0.199, 11.145],
    "enable_physics": False,
    "enable_gfx_replay_save": False,
    "physics_config_file": "./data/default.physics_config.json",
    "num_objects": 10,
    "test_object_index": 0,
    "frustum_culling": True,
}

# build SimulatorConfiguration
def make_cfg(settings):
    sim_cfg = habitat_sim.SimulatorConfiguration()
    if "scene_dataset_config_file" in settings:
        sim_cfg.scene_dataset_config_file = settings[
            "scene_dataset_config_file"
        ]
    sim_cfg.frustum_culling = settings.get("frustum_culling", False)
    if "enable_physics" in settings:
        sim_cfg.enable_physics = settings["enable_physics"]
    if "physics_config_file" in settings:
        sim_cfg.physics_config_file = settings["physics_config_file"]
    if not settings["silent"]:
        print("sim_cfg.physics_config_file = " + sim_cfg.physics_config_file)
    if "scene_light_setup" in settings:
        sim_cfg.scene_light_setup = settings["scene_light_setup"]
    sim_cfg.gpu_device_id = 0
    if not hasattr(sim_cfg, "scene_id"):
        raise RuntimeError(
            "Error: Please upgrade habitat-sim. SimulatorConfig API version mismatch"
        )
    sim_cfg.scene_id = settings["scene"]

    # define default sensor parameters (see src/esp/Sensor/Sensor.h)
    sensor_specs = []

    def create_camera_spec(**kw_args):
        camera_sensor_spec = habitat_sim.CameraSensorSpec()
        camera_sensor_spec.sensor_type = habitat_sim.SensorType.COLOR
        camera_sensor_spec.resolution = [settings["height"], settings["width"]]
        camera_sensor_spec.position = [0.0, settings["sensor_height"], 0.0]
        for k in kw_args:
            setattr(camera_sensor_spec, k, kw_args[k])
        return camera_sensor_spec

    if settings["color_sensor"]:
        color_sensor_spec = create_camera_spec(
            uuid="color_sensor",
            hfov=settings["hfov"],
            sensor_type=habitat_sim.SensorType.COLOR,
            sensor_subtype=habitat_sim.SensorSubType.PINHOLE,
        )
        sensor_specs.append(color_sensor_spec)

    if settings["depth_sensor"]:
        depth_sensor_spec = create_camera_spec(
            uuid="depth_sensor",
            hfov=settings["hfov"],
            sensor_type=habitat_sim.SensorType.DEPTH,
            channels=1,
            sensor_subtype=habitat_sim.SensorSubType.PINHOLE,
        )
        sensor_specs.append(depth_sensor_spec)

    if settings["semantic_sensor"]:
        semantic_sensor_spec = create_camera_spec(
            uuid="semantic_sensor",
            hfov=settings["hfov"],
            sensor_type=habitat_sim.SensorType.SEMANTIC,
            channels=1,
            sensor_subtype=habitat_sim.SensorSubType.PINHOLE,
        )
        sensor_specs.append(semantic_sensor_spec)

    # create agent specifications
    agent_cfg = habitat_sim.agent.AgentConfiguration()
    # sensor_specs: color + depth + semantic sensors
    agent_cfg.sensor_specifications = sensor_specs
    agent_cfg.action_space = {
        "move_forward": habitat_sim.agent.ActionSpec(
            "move_forward", habitat_sim.agent.ActuationSpec(amount=0.25)
        ),
        "turn_left": habitat_sim.agent.ActionSpec(
            "turn_left", habitat_sim.agent.ActuationSpec(amount=10.0)
        ),
        "turn_right": habitat_sim.agent.ActionSpec(
            "turn_right", habitat_sim.agent.ActuationSpec(amount=10.0)
        ),
    }

    # override action space to no-op to test physics
    if sim_cfg.enable_physics:
        agent_cfg.action_space = {
            "move_forward": habitat_sim.agent.ActionSpec(
                "move_forward", habitat_sim.agent.ActuationSpec(amount=0.0)
            )
        }

    return habitat_sim.Configuration(sim_cfg, [agent_cfg])

def simulate(sim, dt, get_observations=False):
    r"""Runs physics simulation at 60FPS for a given duration (dt) optionally collecting and returning sensor observations."""
    observations = []
    target_time = sim.get_world_time() + dt
    while sim.get_world_time() < target_time:
        sim.step_physics(0.1 / 60.0)
        if get_observations:
            observations.append(sim.get_sensor_observations())
    return observations

def test_humanoid_seqpose_controller(humanoid_name, scene_path=None):
    """Test the humanoid controller"""

    # loading the physical scene
    produce_debug_video = True
    num_steps = 1000
    cfg_settings = default_sim_settings.copy()
    if scene_path:
        cfg_settings["scene"] = scene_path
    cfg_settings["enable_physics"] = True

    observations = []
    hab_cfg = make_cfg(cfg_settings)
    with habitat_sim.Simulator(hab_cfg) as sim:
        obj_template_mgr = sim.get_object_template_manager()
        rigid_obj_mgr = sim.get_rigid_object_manager()

        # setup the camera for debug video (looking at 0,0,0)
        sim.agents[0].scene_node.translation = [0.0, 0.0, 2.0]

        # add a ground plane
        cube_handle = obj_template_mgr.get_template_handles("cubeSolid")[0]
        cube_template_cpy = obj_template_mgr.get_template_by_handle(
            cube_handle
        )
        cube_template_cpy.scale = np.array([5.0, 0.2, 5.0])
        obj_template_mgr.register_template(cube_template_cpy)
        ground_plane = rigid_obj_mgr.add_object_by_template_handle(cube_handle)
        ground_plane.translation = [0.0, -0.2, 0.0]
        ground_plane.motion_type = habitat_sim.physics.MotionType.STATIC

        # compute a navmesh on the ground plane
        navmesh_settings = habitat_sim.NavMeshSettings()
        navmesh_settings.set_defaults()
        navmesh_settings.include_static_objects = True
        sim.recompute_navmesh(sim.pathfinder, navmesh_settings)
        sim.navmesh_visualization = True

        # add the humanoid to the world via the wrapper
        humanoid_path = f"data/humanoids/humanoid_data/{humanoid_name}/{humanoid_name}.urdf"
        walk_pose_path = f"data/humanoids/humanoid_data/{humanoid_name}/{humanoid_name}_motion_data_smplx.pkl"

        agent_config = DictConfig(
            {
                "articulated_agent_urdf": humanoid_path,
                "motion_data_path": walk_pose_path,
            }
        )
        if not osp.exists(humanoid_path):
            pytest.skip(f"No humanoid file {humanoid_path}")
        kin_humanoid = kinematic_humanoid.KinematicHumanoid(agent_config, sim)
        kin_humanoid.reconfigure()
        kin_humanoid.update()
        # assert kin_humanoid.get_robot_sim_id() == 1  # 0 is the ground plane

        # set base ground position from navmesh
        # NOTE: because the navmesh floats above the collision geometry we should see a pop/settle with dynamics and no fixed base
        target_base_pos = sim.pathfinder.snap_point(
            kin_humanoid.sim_obj.translation
        )
        kin_humanoid.base_pos = target_base_pos
        assert kin_humanoid.base_pos == target_base_pos
        observations += simulate(sim, 0.01, produce_debug_video)

        # Test controller
        motion_path = (
            "data/humanoids/humanoid_data/walk_motion/CMU_10_04_stageii.pkl"
        )

        if not osp.exists(motion_path):
            pytest.skip(
                f"No motion file {motion_path}. You can create this file by using humanoid_utils.py"
            )
        humanoid_controller = HumanoidSeqPoseController(motion_path)

        base_trans = kin_humanoid.base_transformation
        step_count = 0
        humanoid_controller.reset(base_trans)
        while step_count < num_steps:
            humanoid_controller.calculate_pose()
            humanoid_controller.next_pose(cycle=True)
            new_pose = humanoid_controller.get_pose()

            new_joints = new_pose[:-16]
            new_pos_transform_base = new_pose[-16:]
            new_pos_transform_offset = new_pose[-32:-16]

            # When the array is all 0, this indicates we are not setting
            # the human joint
            if np.array(new_pos_transform_offset).sum() != 0:
                vecs_base = [
                    mn.Vector4(new_pos_transform_base[i * 4 : (i + 1) * 4])
                    for i in range(4)
                ]
                vecs_offset = [
                    mn.Vector4(new_pos_transform_offset[i * 4 : (i + 1) * 4])
                    for i in range(4)
                ]
                new_transform_offset = mn.Matrix4(*vecs_offset)
                new_transform_base = mn.Matrix4(*vecs_base)
                kin_humanoid.set_joint_transform(
                    new_joints, new_transform_offset, new_transform_base
                )
                observations += simulate(sim, 0.0001, produce_debug_video)

            step_count += 1

        # produce some test debug video
        if produce_debug_video:
            from habitat_sim.utils import viz_utils as vut

            motion_name = "test_humanoid_seqpose_controller"
            video_name = (
                os.path.join(output_dir, "scene_video_")
                + motion_name
            )
            vut.make_video(
                observations,
                "color_sensor",
                "color",
                video_name,
                open_vid=False,
            )

if __name__ == "__main__":

    humanoid_name = "female_2"
    scene_path = "bedroom_w_light.glb"
    test_humanoid_seqpose_controller(humanoid_name, scene_path)

[aclegg3]

Hey @zoeyliu1999,

I agree this looks like a simple axis alignment issue. Habitat is Y-up and -Z forward. When I open your .glb asset in Blender, I see it is aligned with Zup. If I re-export from Blender with the "+Y up" flag unchecked I get a glb which will load into Habitat correctly by default. (correct_mesh.zip)

Alternatively, you could also fix this by using the "up" and "front" vector fields of a stage config if, for example, you could not modify the asset itself for some reason.

[zoeyliu1999]

Hey @aclegg3, thank you for your kind reply!

Regarding the "up" and "front" vector fields of a stage config, do I need to construct an entire dataset with all json file equipped for this? As you can see, I currently only generate a single scene .glb file without any config file.

I kind of prefer not to modify the asset, as importing it into Blender for rotation may cause some furniture textures to go missing, and I am unsure if I can correct this error. However, I am also concerned that constructing a complete dataset with all necessary json files might require more effort. Could you please provide some advice? Thanks!

[aclegg3]

Hey @zoeyliu1999 you don't need to construct a complete dataset. For this use case, you should be able to get away with only the stage_config.json file. Just set the scene_id or settings["scene"] to point to that config file instead of the .glb.

The minimal contents of a stage config include only the render asset path (relative to the stage config file). Then the "up" and "front" can be adjusted.

This one works to re-orient the scene for me (in the same directory): bedroom_w_light.stage_config.json

Contents for reference:

{
    "render_asset": "bedroom_w_light.glb",
    "up":[0,1,0],
    "front":[0,0,-1],
    "requires_lighting":true
}

[zoeyliu1999]

Hey hero @aclegg3, it works for me! Thank you so much.

Although I still need to reset the position of the humanoid and scene, the rotation problem is happily solved. Really appreciate your help!

[aclegg3]

No problem, glad it's working for you. Feel free to open new issues if you encounter other challenges and best of luck on your goals.