search

haosulab / SAPIEN

SAPIEN Embodied AI Platform
https://sapien.ucsd.edu/
MIT License
427 stars 41 forks source link

position control #152

Open SshiJwu opened 7 months ago

SshiJwu commented 7 months ago

Hi, I want to execute position control in SAPIEN. But it's not quite what I thought. This is my code: def step(self, action: np.ndarray):

    self.robot.set_drive_target(action)

    print("action:",action)

    qf = self.robot.compute_passive_force(gravity=True, coriolis_and_centrifugal=True)

    self.robot.set_qf(qf)

    for i in range(self.frame_skip):
        self.scene.step()
        self.render()

    self.current_step += 1

    print("joint_position", self.robot.get_qpos())
    print("\n")

    obs = self.get_observation()
    return obs

The returned results are:

action: [ 1.0080942 0.07700591 -0.26779976 0.45527703 -2.4566367 -0.02794247 -0.41149104 -0.28804654 -0.12245759 0.21643543 0.5214863 0.92012674 0.00890571 0.45191255 0.72066396 0.6122276 -0.14939168 0.5470857 0.614574 0.7317573 0.09097223 -0.34906584 0.5128116 0.5564713 1.4659702 0.65489566 0.70423144 0.0875565 -0.29759088 0.29122272] [2024-03-29 21:38:32.082] [SAPIEN] [warning] external force is deprecated and ignored in passive force computation. joint_position [ 5.3307290e+00 -2.7980363e+00 -2.2715263e+00 5.7231402e+00 5.9122033e+00 2.4233885e+00 -5.1686323e-01 -5.9985149e-01 3.4906119e-01 -2.6257661e-01 -9.9840690e-04 2.6208350e-01 2.6045841e-01 1.4904318e+00 1.5557226e+00 1.3111534e+00 -3.4904963e-01 -2.6334012e-01 -2.0598492e-03 7.4446361e-06 5.1040280e-01 -3.3417100e-01 1.5658530e+00 1.5584077e+00 1.3176466e+00 -1.0141965e+00 7.2087932e-01 2.0217310e-01 -6.6974652e-01 -2.5934604e-01]

action: [ 1.0077728 0.21889764 -0.7015863 0.39141732 -1.9953026 1.8720453 -0.38592446 -0.19408308 -0.15859017 0.06714711 0.4636697 0.6357209 0.05944189 0.25535285 0.6238109 0.29917058 -0.19588962 0.45438582 0.5641642 0.51035583 0.13572612 -0.34906584 0.4169842 0.3803882 1.4329351 0.6368792 0.9655694 0.12643999 -0.4998034 0.32632107] [2024-03-29 21:38:33.266] [SAPIEN] [warning] external force is deprecated and ignored in passive force computation. joint_position [ 5.2683725e+00 -2.7284687e+00 -2.2209368e+00 5.5126810e+00 5.5400486e+00 3.0390871e+00 -5.2358037e-01 -5.0338590e-01 3.4906495e-01 -2.4317463e-01 1.3675858e-02 2.6462221e-01 2.5787476e-01 1.5027406e+00 1.5534712e+00 1.3072052e+00 -3.4906074e-01 -2.5425181e-01 6.3799964e-03 9.5727848e-04 5.1789987e-01 -3.3145249e-01 1.5706725e+00 1.5550016e+00 1.3148304e+00 -1.0055885e+00 7.2721928e-01 2.0141706e-01 -6.5943998e-01 -2.5790009e-01]

action: [ 1.0283355 0.25253054 -0.6489771 0.42385158 -2.0095935 1.8318564 -0.35773814 -0.14250918 -0.15065043 0.11706779 0.42541528 0.623275 0.03811157 0.3041759 0.635507 0.30414644 -0.16541882 0.44699684 0.5534656 0.5770409 0.10752289 -0.34906584 0.38562608 0.39418247 1.4362597 0.65269834 0.9621447 0.146154 -0.50012773 0.3357537 ] [2024-03-29 21:38:34.417] [SAPIEN] [warning] external force is deprecated and ignored in passive force computation. joint_position [ 5.2101369e+00 -2.6591597e+00 -2.1719723e+00 5.3027258e+00 5.1823421e+00 3.6911435e+00 -5.2360207e-01 -5.1060206e-01 3.4906504e-01 -2.5150260e-01 8.2643982e-03 2.6403001e-01 2.5711003e-01 1.4986508e+00 1.5507315e+00 1.3075269e+00 -3.4906173e-01 -2.6233345e-01 -6.7510048e-04 -3.0399292e-07 5.1571220e-01 -3.3028448e-01 1.5631597e+00 1.5541407e+00 1.3159699e+00 -1.0027348e+00 7.2851586e-01 2.0109747e-01 -6.5709531e-01 -2.5787556e-01]

How do I make the robot reach the target position?

fbxiang commented 7 months ago

I believe the issue is that set_qf with passive force needs to be called every step to compensate gravity. In SPAIEN 3 (currently dev releases), we provide a better way to achieve this by disabling gravity of the robot completely.

SshiJwu commented 7 months ago

How do I fix it?

fbxiang commented 7 months ago

You can try set_qf every step.

    self.robot.set_drive_target(action)

    print("action:",action)

    qf = self.robot.compute_passive_force(gravity=True, coriolis_and_centrifugal=True)
    self.robot.set_qf(qf)

    for i in range(self.frame_skip):

        # add these lines
        qf = self.robot.compute_passive_force(gravity=True, coriolis_and_centrifugal=True)
        self.robot.set_qf(qf)

        self.scene.step()
        self.render()

    self.current_step += 1

    print("joint_position", self.robot.get_qpos())
    print("\n")

    obs = self.get_observation()
    return obs
SshiJwu commented 7 months ago

Thanks for your reply! I have one more question: How can I get a GT point cloud of objects in the scene?

fbxiang commented 7 months ago

All geometries in SAPIEN are mesh so there is no GT point cloud. I guess you could mean 2 things.

  1. Point cloud from GT depth map of cameras. This has been discussed in SAPIEN's documentation rendering section.

  2. Sampled point cloud from collision or visual meshes. For this I would suggest converting SAPIEN meshes to trimesh and use trimesh sampling functions. Assuming you are using SAPIEN 2, the example code for this conversion can be found here https://github.com/haosulab/ManiSkill2/blob/main/mani_skill2%2Futils%2Ftrimesh_utils.py

SshiJwu commented 7 months ago

How do I fix it?

Traceback (most recent call last): File "examples/gen_demonstration_expert.py", line 236, in main() File "examples/gen_demonstration_expert.py", line 75, in main env = create_env(task_name=task_name, File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/create_env.py", line 37, in create_env env = LaptopRLEnv(env_params, friction=5) File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/rl_env/laptop_env.py", line 20, in init super().init(use_gui, frame_skip, friction=friction, index=index, renderer_kwargs) File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/sim_env/laptop_env.py", line 15, in init super().init(use_gui=use_gui, frame_skip=frame_skip, renderer_kwargs) File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/rl_env/base.py", line 32, in init super().init(use_gui=use_gui, frame_skip=frame_skip, use_visual_obs=use_visual_obs, renderer_kwargs) File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/sim_env/base.py", line 25, in init engine, renderer = get_engine_and_renderer(use_gui=use_gui, need_offscreen_render=need_offscreen_render, File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/sim_env/constructor.py", line 36, in get_engine_and_renderer _renderer = sapien.SapienRenderer(default_mipmap_levels=mipmap_levels, offscreen_only=not use_gui, RuntimeError: Cannot find any cuda device suitable for rendering.

I run nvidia-smi,nvcc -Vand vulkaninfo,then return no error. I think this is a error about SAPIEN.

fbxiang commented 7 months ago

Can you start a new issue and post the result of vulkaninfo and nvidia-smi?