Adding New Gripper Problem

[Huangruopeng]

Hi

Thank you for your work, this repo is great.

Now I trying to add my own robotiq 85 gripper. After generate .usd file. I meet this problem.

2024-07-20 04:22:10 [5,013ms] [Error] [omni.physx.plugin] PhysX error: PxRigidDynamic::setGlobalPose: pose is not valid., FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpRigidDynamic.cpp, LINE 91 2024-07-20 04:22:10 [5,013ms] [Error] [omni.physx.plugin] PhysX error: PxRigidDynamic::setLinearVelocity: velocity is not valid., FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpRigidDynamic.cpp, LINE 207 2024-07-20 04:22:10 [5,013ms] [Error] [omni.physx.plugin] PhysX error: PxRigidDynamic::setAngularVelocity: velocity is not valid., FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpRigidDynamic.cpp, LINE 231

I try to set different joint velocity, But still has this problem.

Thank you so much!

[kninad]

This looks like this is an Isaac Sim gripper import issue. Could you share your URDF for the gripper (along with the meshes etc) so that we can also give a try at our end. Tagging @FelipeCasas!

[FelipeCasas]

Hello,

Could you expand more about your problem, did the problem arise when running our standalone with your gripper or does this problem arise when importing the gripper .urdf with the GUI?

Kind Regards

[Huangruopeng]

Hi,

I try to run this command:

lm-2023@lm2023-XPS-8950:~/.local/share/ov/pkg/isaac-sim-2023.1.0-hotfix.1$ ./python.sh /home/lm-2023/Isaac_Sim/grasping/isaac_sim_grasping/visualize_grasps.py --json_dir=/home/lm-2023/Isaac_Sim/grasping/Dataset/graspit_grasps/robotiq --gripper_dir=/home/lm-2023/Isaac_Sim/grasping/isaac_sim_grasping/grippers --objects_dir=/home/lm-2023/Isaac_Sim/grasping/Dataset/google_objects_usd --num_w=10 --ub=3 --lb=0 --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

And the isaacsim will be launched, but just keep 2 seconds, then it closed, and my gripper robotiq 85 2 fingers can be displayed in the interface, but just 2 senconds.

The error info:

This is my URDF file: robotiq_2finger.zip

I follow this setting to import my urdf:

Because my gripper has 8 dofs, so I just use repo graspit Barrett-025.json file to test it can be run. (Maybe the dofs data is not match! causing some errs)

Thank you so much! Kind Regards!

[Huangruopeng]

Also, I generated my own json file using graspit using neuralgrasps-dataset-generation repo and noticed that the json file generated using it does not match the format of the json I downloaded from the dataset folder provided in this repo, are there any modifications I need to make to the json I generated for my robotiq 85 2f gripper?

Thank you so much for your help!

[FelipeCasas]

I see.., within the managers.py file there are many manager classes, since you are using the visualize grasp standalone you will need to add your gripper information to the dictionaries of that class (V_Manager). Try doing so in the https://github.com/IRVLUTD/isaac_sim_grasping/blob/6395a648574935ccb74aa3dd18eaeb0f36b274ec/managers.py#L817 dictionaries and let me know what happens.

Thanks for letting us now about this issue! We will most likely change the dictionaries to an external file so they can be easily changed and are not so repetitive.

[Huangruopeng]

I also modify this part:

[FelipeCasas]

Okay, would you mind sharing your .json file and the managers.py file so we can recreate the errors?

[Huangruopeng]

.json file Barrett-025_mug.json

managers.py managers.zip

[kninad]

Also, I generated my own json file using graspit using neuralgrasps-dataset-generation repo and noticed that the json file generated using it does not match the format of the json I downloaded from the dataset folder provided in this repo, are there any modifications I need to make to the json I generated for my robotiq 85 2f gripper?

Thank you so much for your help!

I guess the only change would in the grasp pose orientation. For isaac sim grasp poses iirc the grasp pose is a 7D vector: (posn, quat_wxyz) where the quaternion is in wxyz format whereas for the neuralgrasps-dataset-generation, graspit might save it in (posn, quat_xyzw) format. So try changing the graspit pose quaternion to wxyz format before applying in isaac sim? Perhaps that should at least give you the gripper in correct orientation.

[FelipeCasas]

Thanks! I will try to reproduce your error. In the meantime, you can try what @kninad says. Isaac Sim quaternion convention is wxyz.

[Huangruopeng]

Thank you for your help, just a small point maybe is important for you reproduce. My urdf read mesh file the path is based on my own path. Please modify it to your own path. <?xml version="1.0" ?>

filename = should be change to your path. Thank you so much!

[Huangruopeng]

Now I can stable visualize it by deleting collision part in my urdf and generate to my .usd without collisions part ( Because I check your .usd file in this repo other gripper don't have collision)

But Now my gripper can't close and open normally, I guess is dofs parameters is not right, considering my gripper also has mimic joints how should I factor this in, any tips would be really helpful. Can you give some instruction where I can get the right open and close status dofs data?

Close status:

Open status:

[Huangruopeng]

Also, I generated my own json file using graspit using neuralgrasps-dataset-generation repo and noticed that the json file generated using it does not match the format of the json I downloaded from the dataset folder provided in this repo, are there any modifications I need to make to the json I generated for my robotiq 85 2f gripper? Thank you so much for your help!

I guess the only change would in the grasp pose orientation. For isaac sim grasp poses iirc the grasp pose is a 7D vector: (posn, quat_wxyz) where the quaternion is in wxyz format whereas for the neuralgrasps-dataset-generation, graspit might save it in (posn, quat_xyzw) format. So try changing the graspit pose quaternion to wxyz format before applying in isaac sim? Perhaps that should at least give you the gripper in correct orientation.

I noticed there are some other parameters like graspit_dofs, final_dofs and fall time in the graspit_grasps json in your dataset while in my json there is only poses, dofs, quality and contact links (the last two are not in your json) am I doing something wrong/missing something?

Thank you!

[FelipeCasas]

Hello, I was able to replicate your error. It seems that your gripper .usd is not functioning with the PhysiX software. I would recommend to use Isaac Sim GUI to change the .usd of your gripper to a working version. You can use our gripper .usd's as reference.

You should be able to manually change the joint positions through the GUI without generating PhysiX errors. An example is below:

[FelipeCasas]

Now I can stable visualize it by deleting collision part in my urdf and generate to my .usd without collisions part ( Because I check your .usd file in this repo other gripper don't have collision)

But Now my gripper can't close and open normally, I guess is dofs parameters is not right, considering my gripper also has mimic joints how should I factor this in, any tips would be really helpful. Can you give some instruction where I can get the right open and close status dofs data?

Close status:

For the open positions you must go to your gripper .usd and inspect the joints this way you can set the initial positions when filtering transferred grasps. The close_dir is a mask that specifies which dofs to close in order to perform the grasp and with which dofs to exert force once the grasp is secured so that the object doesn't fall.

I hope all this information helps! Let us know if other problems arise.

[kninad]

Also, I generated my own json file using graspit using neuralgrasps-dataset-generation repo and noticed that the json file generated using it does not match the format of the json I downloaded from the dataset folder provided in this repo, are there any modifications I need to make to the json I generated for my robotiq 85 2f gripper? Thank you so much for your help!

I guess the only change would in the grasp pose orientation. For isaac sim grasp poses iirc the grasp pose is a 7D vector: (posn, quat_wxyz) where the quaternion is in wxyz format whereas for the neuralgrasps-dataset-generation, graspit might save it in (posn, quat_xyzw) format. So try changing the graspit pose quaternion to wxyz format before applying in isaac sim? Perhaps that should at least give you the gripper in correct orientation.

I noticed there are some other parameters like graspit_dofs, final_dofs and fall time in the graspit_grasps json in your dataset while in my json there is only poses, dofs, quality and contact links (the last two are not in your json) am I doing something wrong/missing something?

Thank you!

In the shared dataset, the values in the dict are after simulating with isaac sim. So we record the fields of interest from isaac simulation (final_dofs, fall_time). Whereas with the org. graspit generation, only the pose, dofs and quality are of interest (you can safely ignore contact_links). The json from graspit is actually read by our isaacsim testing script here!

[Huangruopeng]

Hello, I was able to replicate your error. It seems that your gripper .usd is not functioning with the PhysiX software. I would recommend to use Isaac Sim GUI to change the .usd of your gripper to a working version. You can use our gripper .usd's as reference.

You should be able to manually change the joint positions through the GUI without generating PhysiX errors. An example is below:

Thank you for your advice! But Now if I modify the .usd file. the instansbale.usd file will be unmatched. So can I generate instansable.usd file from .usd file?

Thank you so much!

[FelipeCasas]

But Now if I modify the .usd file. the instansbale.usd file will be unmatched. So can I generate instansable.usd file from .usd file?

The instanceable meshes were used to reduce the memory usage of the simulation when running many workstations and many standalones in parallel. If you would like to use them, you will need to modify both .usd files, else you can import the gripper without the "Create Instanceable Asset" option and modify that one.

[Huangruopeng]

Thank you for your reply!

Now I have add my robotiq 85 gripper, and it can close and open normally. But When I run this : ./python.sh /home/lm-2023/Isaac_Sim/grasping/isaac_sim_grasping/standalone.py --json_dir=/home/lm-2023/Isaac_Sim/grasping/Dataset/graspit_grasps/robotiq --gripper_dir=/home/lm-2023/Isaac_Sim/grasping/isaac_sim_grasping/grippers --objects_dir=/home/lm-2023/Isaac_Sim/grasping/Dataset/google_objects_usd --output_dir=/home/lm-2023/Isaac_Sim/grasping/ --num_w=2 --test_time=3 --controller=position --print_results --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

The object and gripper can't contact each other with collision. And the object can't drop.

So where I need to modify? .usd file or some scripts?

Thank you so much!

[Huangruopeng]

Hi I also try h5_hand example, and set over-bound joint data. It also shows overlap. So, if I need to modify some code? Thank you so much!

[kninad]

@FelipeCasas maybe its an issue with the collision meshes in the usd?

[FelipeCasas]

Hello, it's nice to know you were able to fix your problem with controlling the robot!

Our own gripper .usds can be misleading, as in the h5_hand, some only have a visuals frame with the mesh. If you inspect the properties on the visuals you will find a collider property within it. Make sure that your gripper has the colliders set up for the meshes that make it up.

I also try h5_hand example, and set over-bound joint data. It also shows overlap.

Some grippers also have self-collision turned off, you can make them overlap provided the correct joint positions. You would likely need to set this up with your gripper; if it is turned on, the physics simulator will try to process the collisions between your gripper links and unexpected behavior may rise when loading or moving your gripper.