Difficulties with sim-to-real applications

[Victeldo]

I'm working with the Kinova gen 3 arm and I'm trying to train a OSC Pose (Fixed Impedance) policy in robosuite before applying it to the real robot.

When testing in Gazebo and passing in the robosuite actions as cartesian poses, I'm seeing a discrepancy in the actions for the robosuite simulator and the Kinova simulator. Namely, the magnitude of the actions in robosuite are much smaller than the actions in the real robot (where the real robot fully extends within 50 or so actions while the robot in robosuite moves that much in 500 or so actions).

Any advice on what to do here would be appreciated. For reference, I'm using https://github.com/Kinovarobotics/ros_kortex/ to pass the robosuite actions for the sim-to-real.

[kevin-thankyou-lin]

Sounds like you'll want to do some system indentification for the robot.

Some knobs to tune:

• OSC controller gains e.g. in robosuite/controllers/config/osc_pose.json
• Joint damping terms in robot xml e.g. robosuite/models/assets/robots/kinova3/robot.xml
• Potentially the control frequency in robosuite (lower freq. = more time allotted per env step)

[Victeldo]

When passing the positions of the OSC Pose as 0.01 of what the robosuite policy produces, the actions appear to be much more accurate (leading me to believe that robosuite operates in terms of centimeters while the real robot operates with meters for Cartesian space).

With respect to the control frequency, reducing it didn't do much (with only a fraction of the robosuite commands registering in the real robot) -- which may have something to do with the robot itself.

[kevin-thankyou-lin]

Gotcha, sounds like you've resolved your issue?

[Victeldo]

Hi Kevin,

I did, ends up that Kortex takes in actions in terms of degrees and cm. With respect to the control frequency, I was able to improve the frequency of actions registering (but every now and then an action doesn't register).

[kevin-thankyou-lin]

Great! What do you mean the action doesn't register?

[Victeldo]

For example, when I use teleoperation, when I tap on a key three times, it jerks 3 times in simulation. However, on the real robot, it would sometimes only jerk once or twice (and not all three times). This happens a lot more with position changes as opposed to orientation changes.

[kevin-thankyou-lin]

Oh weird, unfortunately, I mightn’t be able to help too much with this aspect as it seems to be related to the real robot itself/the communication/networking system

On Mon, 26 Aug 2024 at 13:42, Prithvi @.***> wrote:

For example, when I use teleoperation, when I tap on a key three times, it jerks 3 times in simulation. However, on the real robot, it would sometimes only jerk once or twice (and not all three times). This happens a lot more with position changes as opposed to orientation changes.

— Reply to this email directly, view it on GitHub https://github.com/ARISE-Initiative/robosuite/issues/502#issuecomment-2310836762, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH6MY6LREXP5EJN4LZ6FCETZTNZH7AVCNFSM6AAAAABLWT4BCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJQHAZTMNZWGI . You are receiving this because you commented.Message ID: @.***>

[Victeldo]

That makes sense. Thank you for your help otherwise! I'll close this issue as the remaining problem is not related to Robosuite.

[kevin-thankyou-lin]

Happy to help, and good luck!