IndustReal 16mm Peg Simulation

[patricknaughton01]

Hello,

Thanks for the great paper on IndustReal, it's very exciting to see techniques like reinforcement learning working on these contact-rich tasks. I'm using Isaac Gym and specifically the IndustReal peg insertion environments to simulate some simple hand-designed controllers for industrial peg insertion. I was able to get the Franka arm to follow some position waypoints with the provided task-space impedance controller and successfully insert most pegs, but I am running into a strange issue with the 16mm round peg. As the gripper closes on the peg, something seems to jerk the end effector, and after grasping the peg, when trying to follow the insertion trajectory, the arm instead veers off to one side. For some reason this only seems to happen with the 16mm round peg (tested with the 8mm and 12mm pegs as well as the 16mm rectangular peg). Any ideas on what could be going on here?

I've included videos of the observed trajectories with the 16mm peg to show the veering behavior and one of the 12mm peg to show the intended behavior. The only thing changed between the two videos was the IndustRealEnvPegs.yaml file to specify the desired subassembly. The red line shows the planned trajectory for the gripper midpoint in the first environment and the green lines show its actual trajectory. The random motion at the end of the videos is due to the uninitialized policy networks taking over control of the arm.

https://github.com/NVIDIA-Omniverse/IsaacGymEnvs/assets/28968518/daeac62a-aee9-4b29-95be-aded60186fc5

https://github.com/NVIDIA-Omniverse/IsaacGymEnvs/assets/28968518/85ca7138-dacb-4068-8285-4102121ab4ed

Thanks again for the great paper and making the code public!

[bingjietang718]

Hi,

Thank you so much for your interest in IndustReal. Seems like the videos can't be played because the files are corrupted. Can you maybe re-upload the videos?

Best, Bingjie

[patricknaughton01]

Sorry about that, do these videos show properly?

https://github.com/NVIDIA-Omniverse/IsaacGymEnvs/assets/28968518/3a2e1094-700d-4ad6-b906-aa385c6e5dcf

https://github.com/NVIDIA-Omniverse/IsaacGymEnvs/assets/28968518/5e8761b4-a031-4a40-890a-e7c1b35845fb

I also tried removing the self.close_gripper(sim_steps=self.cfg_task.env.num_gripper_close_sim_steps) from the reset_idx function, so that the plug just falls out of the gripper before it can close on it. With this modification, the gripper seems to follow a reasonable insertion trajectory (but without holding the plug of course). The plug tends to either fall onto the table or balance on the face of the socket, but doesn't move of its own accord. I guess the strange simulation is due to a weird interaction between the plug and the gripper? Thanks for your help.

[bingjietang718]

Hi Patrick,

Can you try out the solution mentioned in this answer?

Best, Bingjie

[patricknaughton01]

Hi Bingjie,

Thanks for your help, that solution seems to have worked. I was originally trying the solution mentioned in this answer; I guess the 16mm peg might be introducing a more significant deviation between the midpoint and center points? Thanks again for your help!

Patrick