[Bug Report] SE(3) teleoperation by keyboard crashes on Franka and cannot control UR10

[vhvkhoa]

If you are submitting a bug report, please fill in the following details and use the tag [bug].

Describe the bug

I ran the code with Isaac-Reach-Franka-Play-v0 and receives the following error message:

ValueError: Invalid action shape, expected: 7, received: 6.

I also tried with Isaac-Reach-UR10-Play-v0, Isaac Sim window pops up this time but pressing keyboard keys do not produce any movement to the arm.

Steps to reproduce

The following command is for Franka robot arm:

./isaaclab.sh -p source/standalone/environments/teleoperation/teleop_se3_agent.py --task Isaac-Reach-Franka-Play-v0 --device keyboard

And the following command is for UR10 robot arm:

./isaaclab.sh -p source/standalone/environments/teleoperation/teleop_se3_agent.py --task Isaac-Reach-UR10-Play-v0 --device keyboard

System Info

Describe the characteristic of your environment:

• Isaac Sim Version: 4.0.0
• OS: 22.04
• GPU: A6000
• CUDA: 12.4
• GPU Driver: 550.90.07

Additional context

I tried to investigate the problem with UR10 and figured out there are two problems:

• Seems like keyboard controller in the code intends to update robot arm by updating its relative position, via adding or subtracting a small value to its current pose. However, the robot requires absolute pose.
• I fixed the above problem by simply adding variables that stores the absolute positions. Then I figured out another problem, that is the keyboards actually controls the torque of every joints, rather than position and rotation of the end effector.

With those observation, I can fix the problem of this program. For Franka robot, which is 7 DoF, I just simply need to add one more number to self._rot and a pair of keyboard keys to increase/decrease it, and the problem goes away. However this is just a workaround and the keyboard is supposed to control the end-effector.

Checklist

• [x] I have checked that there is no similar issue in the repo (required)
• [x] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

• [ ] Criteria 1
• [ ] Criteria 2

[Mayankm96]

Please use the IK variants of these environments that control the task-space using the device.

https://isaac-sim.github.io/IsaacLab/source/features/environments.html#manipulation