balakumar-s
commented
11 months ago You would need IsaacSim 2022.2.1 to run the example. We are working on getting the examples to run on 2023.1. I am glad that the sphere tutorial worked in 2023.
Closed cedricgoubard closed 11 months ago
balakumar-s
commented
11 months ago You would need IsaacSim 2022.2.1 to run the example. We are working on getting the examples to run on 2023.1. I am glad that the sphere tutorial worked in 2023.
cedricgoubard
commented
11 months ago Thank you for your reply; I am building the 2022 image now, will let you know if it fixes the issue.
Also, here are a couple of typos I noticed in the tutorial:
curobo/examples/isaac_sim/convert_urdf_to_usd.py should be replaced by curobo/examples/isaac_sim/utils/convert_urdf_to_usd.pycedricgoubard
commented
11 months ago I got past the error with 2022.2.1 :partying_face: I am now having a problem with my yaml file, but I should be able to fix this myself. I'll close the issue when I manage to finish the tutorial.
FYI, I have also identified the problem in 2023.1, it was caused by this line in my Dockerfile:
RUN /isaac-sim/python.sh -m pip install tomli tqdm usd-core wheel cloudpickle farama-notifications jax-jumpy botocore==1.19.52 boto3==1.16.52I had to add this to fix a ModuleNotFoundError: No module named 'tomli' error.
balakumar-s
commented
11 months ago Thanks, yes you do have to install tomli. You actually shouldn't install usd-core as isaac sim comes with it's own usd-core. pip installing usd-core will break isaac sim.
cedricgoubard
commented
11 months ago That is weird; I am still stuck even with version 2022.2.1, but I am not really getting an error message this time... The error is:
...
/isaac-sim/python.sh: line 41: 36 Killed $python_exe "$@" $args
There was an error running pythonI am getting a warning about usd-core at the beginning though:
USDParser failed to import, install curobo with pip install .[usd] or pip install usd-coreTo be clear, this is without installing it with pip.
balakumar-s
commented
11 months ago I think this is because the path to your urdf is not getting loaded correctly in Isaac Sim's urdf importer. Is it possible for me to get access to the urdf folder and the configuration file you generated?
lianghongzhuo
commented
11 months ago Thanks, yes you do have to install tomli. You actually shouldn't install
usd-coreas isaac sim comes with it's ownusd-core. pip installingusd-corewill break isaac sim.
I also mislead by the terminal output and install usd-core mannully using pip. I have to remove isacc sim and reinstall it. Hope that isacc sim can provide a clean python binding that make isacc sim independent. Do you have the opputunity to forward this wish? Thank you!
lianghongzhuo
commented
11 months ago The down side also includes, currently using isacc sim means binding the torch version ,python version to the isaac provided version.
balakumar-s
commented
11 months ago Thanks, yes you do have to install tomli. You actually shouldn't install
usd-coreas isaac sim comes with it's ownusd-core. pip installingusd-corewill break isaac sim.I also mislead by the terminal output and install usd-core mannully using pip. I have to remove isacc sim and reinstall it. Hope that isacc sim can provide a clean python binding that make isacc sim independent. Do you have the opputunity to forward this wish? Thank you!
You can leave feedback for Isaac Sim here: https://forums.developer.nvidia.com/t/feedback-request-from-isaac-sim-users/267919
cedricgoubard
commented
11 months ago Hi @balakumar-s,
I found the error thanks to your feedback; it was a typo in the YAML file (I had declared a sphere with a centre but no radius).
The simulation starts now, but the robot gets immediately stuck and I see a lot of PhysX errors like this one:
2023-11-06 14:59:30 [519,081ms] [Error] [omni.physx.plugin] PhysX error: Illegal BroadPhaseUpdateData
, FILE /buildAgent/work/16dcef52b68a730f/source/lowlevelaabb/src/BpBroadPhaseABP.cpp, LINE 4024
2023-11-06 14:59:30 [519,083ms] [Warning] [omni.physx.plugin] Invalid PhysX transform detected for /World/amiga/castor_back_left.
2023-11-06 14:59:30 [519,083ms] [Warning] [omni.physx.plugin] Invalid PhysX transform detected for /World/amiga/castor_back_right.
2023-11-06 14:59:30 [519,083ms] [Warning] [omni.physx.plugin] Invalid PhysX transform detected for /World/amiga/castor_front_left.
2023-11-06 14:59:30 [519,083ms] [Warning] [omni.physx.plugin] Invalid PhysX transform detected for /World/amiga/castor_front_right.
2023-11-06 14:59:30 [519,083ms] [Warning] [omni.physx.plugin] Invalid PhysX transform detected for /World/amiga/back_wheel_left.
2023-11-06 14:59:30 [519,083ms] [Warning] [omni.physx.plugin] Invalid PhysX transform detected for /World/amiga/back_wheel_right.
2023-11-06 14:59:30 [519,083ms] [Warning] [omni.physx.plugin] Invalid PhysX transform detected for /World/amiga/front_wheel_left.
2023-11-06 14:59:30 [519,083ms] [Warning] [omni.physx.plugin] Invalid PhysX transform detected for /World/amiga/front_wheel_right.The links mentioned are all the ones with passive joints, and this answer says that the error is related to invalid transforms... Should I publish those transforms with ROS? IS there an alternative?
Also, they seem to be colliding with the ground (see below) which makes sense since my root link is located roughly in the centre of the base, but that might be creating issues?
balakumar-s
commented
11 months ago The examples will put the base link at 0,0,0 so that is definitely a problem. You can add an extra link that will have an offset on z axis and make it the root. Have a 5cm gap between the floor and the base.
Regarding passive joints, I'm not sure if they work in sim. You can make them as fixed just to check if everything else works.
cedricgoubard
commented
11 months ago Hi,
I have changed the passive joints to fixed, and I am spawning the robot above the floor (I added position=np.array([0.4, 0.4, 0.4]) to this line).
The arm is still not moving when I move the red cube - I am suppose to do something to make it move other than starting the simulation?
I think this could be linked to 2 things:
base_link or ee_link in my config file, but I would expect some weird movement or error; otherwise, how can I check this?2023-11-07 13:53:10 [33,058ms] [Warning] [omni.physx.plugin] The rigid body at /World/amiga/amiga_arm_base has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for. If you do not want the objects to collide, add colliders regardless then disable the 'enable collision' property.Also, interestingly, the robot does not fall to the floor when I start the simulation...
Any advice would be greatly appreciated, thank you for your help :pray:
balakumar-s
commented
11 months ago I'm not sure if this is the problem but we can check.
Remove this check off velocities being less than threshold here https://github.com/NVlabs/curobo/blob/e0e804e9062da33c13aa262a1da253d364b8768a/examples/isaac_sim/motion_gen_reacher.py#L268
This is used to make sure the robot is static before querying a motion plan. I've noticed that some robots can have non zero velocity at start due to some physics issues. I thought I fixed it but maybe it still exists.
If it still doesn't work, can you run the mpc example? Or the IK example to see if they work for your robot?
cedricgoubard
commented
11 months ago Good morning,
Removing the line you mentioned fixed the problem, the end effector is now following the red cube! I also had to remove the 40cm position offset I added previously, otherwise the end effector was always off by the same amount.
The collision behaviour is looking odd though:
Any idea why?
balakumar-s
commented
11 months ago Regarding position offset, you need to convert the pose from the world frame to the position offset as it's not being done in the example.
Regarding self collisions, did you add the colliding links to the ignore list?
Can you paste your t robot configuration file ?
cedricgoubard
commented
11 months ago I generated the self_collision_ignore list from my previous MoveIt configuration file; it includes links that are both always and never colliding.
In the picture, the links colliding are amiga_arm_upper_arm_link and amiga_gripper_finger_[1 or 2]_link_[probably 2 and 3].
EDIT: I do not think the problem is with my self_collision_ignore; when I comment it and set self_collision_ignore: null, everything is still colliding:
balakumar-s
commented
11 months ago I missed a step in the tutorial.
In your configuration file, can you add the names of all links you want to enable collision checking as a list to collision_link_names ?
cedricgoubard
commented
11 months ago I have added them, and I also had to add self_collision_buffer as a dictionary with each link and 0.02 as value.
I am now getting another error because the gripper joints are not in cspace.joint_names:
Traceback (most recent call last):
File "curobo/examples/isaac_sim/motion_gen_reacher.py", line 325, in <module>
main()
File "curobo/examples/isaac_sim/motion_gen_reacher.py", line 163, in main
velocity_scale=[0.25, 1, 1, 1, 1.0, 1.0, 1.0, 1.0, 1.0],
File "/isaac-sim/kit/python/lib/python3.7/contextlib.py", line 74, in inner
return func(*args, **kwds)
File "/curobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 338, in load_from_robot_config
robot_cfg = RobotConfig.from_dict(robot_cfg, tensor_args)
File "/curobo/curobo/src/curobo/types/robot.py", line 49, in from_dict
CudaRobotGeneratorConfig(**data_dict_in["kinematics"], tensor_args=tensor_args)
File "/curobo/curobo/src/curobo/cuda_robot_model/cuda_robot_model.py", line 128, in from_config
generator = CudaRobotGenerator(config)
File "/curobo/curobo/src/curobo/cuda_robot_model/cuda_robot_generator.py", line 217, in __init__
self.initialize_tensors()
File "/isaac-sim/kit/python/lib/python3.7/contextlib.py", line 74, in inner
return func(*args, **kwds)
File "/curobo/curobo/src/curobo/cuda_robot_model/cuda_robot_generator.py", line 296, in initialize_tensors
self.cspace.inplace_reindex(self.joint_names)
File "/curobo/curobo/src/curobo/cuda_robot_model/types.py", line 128, in inplace_reindex
new_index = [self.joint_names.index(j) for j in joint_names]
File "/curobo/curobo/src/curobo/cuda_robot_model/types.py", line 128, in <listcomp>
new_index = [self.joint_names.index(j) for j in joint_names]
ValueError: 'amiga_gripper_palm_finger_1_joint' is not in listShould I add them? Would that not complicate the planning, since I am only interested in moving the arm?
balakumar-s
commented
11 months ago Add all actuated joints to joint names and then you can add joints to lock as below in the configuration file:
cedricgoubard
commented
11 months ago Thank you; collisions spheres are now appearing, but the plan does not converge. I am guessing this is either because my starting configuration already has collisions, or because my parameters are wrong
cspace:
joint_names:
- amiga_arm_shoulder_pan_joint
- amiga_arm_shoulder_lift_joint
- amiga_arm_elbow_joint
- amiga_arm_wrist_1_joint
- amiga_arm_wrist_2_joint
- amiga_arm_wrist_3_joint
- amiga_gripper_palm_finger_1_joint
- amiga_gripper_finger_1_joint_1
- amiga_gripper_finger_1_joint_2
- amiga_gripper_finger_1_joint_3
- amiga_gripper_palm_finger_2_joint
- amiga_gripper_finger_2_joint_1
- amiga_gripper_finger_2_joint_2
- amiga_gripper_finger_2_joint_3
- amiga_gripper_finger_middle_joint_1
- amiga_gripper_finger_middle_joint_2
- amiga_gripper_finger_middle_joint_3
retract_config: [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00]
null_space_weight: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
cspace_distance_weight: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
max_jerk: 500.0
max_acceleration: 30.0
balakumar-s
commented
11 months ago Does the IK example work? That would not consider the start state and would teleport the robot to a collision-free IK solution.
Also the self collision buffer can all be set to 0.0. They can also be an empty dictionary.
cedricgoubard
commented
11 months ago When I run /isaac-sim/python.sh curobo/examples/isaac_sim/ik_reachability.py --robot /ros_ws/src/curobo_ros/cfg/amiga.yaml --visualize_spheres, the arm does not move, but there is no error message.
It looks like it's spawning in the same position as before.
Also, you might want to fix the yaml file from the tutorial from self_collision_buffer: null to self_collision_buffer: {}; it raised an error before, complaining about self_collision_buffer.keys().
balakumar-s
commented
11 months ago And in the ik_reachability example, moving the target cube doesn't do anything?
I will add a check for empty dictionary in the code. For now null should work.
balakumar-s
commented
11 months ago One way to debug if links are colliding:
collision_link_names tag and see when the plan starts to converge. Then you can debug just the link that is causing the plan to fail. cedricgoubard
commented
11 months ago I followed your advice, it looks like the problem happens because some links' spheres collide with the plane/table obstacle.
I manually added an offset to the robot, the spheres and the target in the example code, and it works! I will refine my spheres and start testing this on the real robot next, but I think I can close this issue now.
Thank you very much for all your help :pray:
balakumar-s
commented
11 months ago Your robot looks cool. Share any videos of your robot (sim or real) working with cuRobo here: https://github.com/NVlabs/curobo/discussions/categories/community-demos
Thanks for using the tutorial and catching bugs. I have updated the tutorial to incorporating the changes from your use.
Good morning,
I wanted to test curobo with a robot we are using (ur10e arm with a gripper on a mobile base). I followed the tutorial and everything worked until the last test, when I run
omni_python curobo/examples/isaac_sim/motion_gen_reacher.py --robot /{PATH}/ur5e.yml --visualize_spheresThis might be related with the IsaacSim setup I am using:
nvidia/cudagl:11.3.1-devel-ubuntu20.04as a base image so thatnvccis accessible to build curobo and nvbloxAs mentioned, everything builds correctly, and I am able to run IsaacSim to generate the USD and the spheres. Here is the yaml file, and here is full log of the error I get:
Full Log
``` # /isaac-sim/python.sh curobo/examples/isaac_sim/motion_gen_reacher.py --robot /root/ros_ws/src/curobo_ros/cfg/amiga.yaml --visualize_spheres [Warning] [omni.isaac.kit.simulation_app] Modules: ['pxr', 'pxr.Tf._tf', 'pxr.Tf', 'pxr.Trace._trace', 'pxr.Trace', 'pxr.Work._work', 'pxr.Work', 'pxr.Plug._plug', 'pxr.Plug', 'pxr.Gf._gf', 'pxr.Gf', 'pxr.Vt._vt', 'pxr.Vt', 'pxr.Ar._ar', 'pxr.Ar', 'pxr.Kind._kind', 'pxr.Kind', 'pxr.Sdf._sdf', 'pxr.Sdf', 'pxr.Pcp._pcp', 'pxr.Pcp', 'pxr.Usd._usd', 'pxr.Usd', 'pxr.UsdGeom._usdGeom', 'pxr.UsdGeom', 'pxr.Ndr._ndr', 'pxr.Ndr', 'pxr.Sdr._sdr', 'pxr.Sdr', 'pxr.UsdShade._usdShade', 'pxr.UsdShade', 'pxr.UsdPhysics._usdPhysics', 'pxr.UsdPhysics'] were loaded before SimulationApp was started and might not be loaded correctly. [Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...) [Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/2023.1/kit_20231102_114656.log 2023-11-02 11:46:56 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available. [0.137s] [ext: omni.kit.async_engine-0.0.0] startup [1.409s] [ext: omni.activity.core-1.0.1] startup [1.418s] [ext: omni.assets.plugins-0.0.0] startup [1.423s] [ext: omni.stats-0.0.0] startup [1.426s] [ext: omni.client-1.0.1] startup [1.453s] [ext: omni.activity.profiler-1.0.2] startup [1.458s] [ext: omni.gpu_foundation-0.0.0] startup [1.502s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup [1.505s] [ext: carb.windowing.plugins-1.0.0] startup 2023-11-02 11:46:58 [2,064ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored): Corsair CORSAIR K60 RGB PRO Low Profile Mechanical Gaming Keyboard [030000001c1b0000ad1b000011010000] [2.102s] [ext: omni.kit.renderer.init-0.0.0] startup |---------------------------------------------------------------------------------------------| | Driver Version: 535.113.01 | Graphics API: Vulkan |=============================================================================================| | GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID | | | | | | | Device-ID | UUID | | | | | | | Bus-ID | | |---------------------------------------------------------------------------------------------| | 0 | NVIDIA GeForce RTX 3060 | Yes: 0 | | 12534 MB | 10de | 0 | | | | | | | 2503 | f55b400e.. | | | | | | | 65 | | |=============================================================================================| | OS: 20.04.4 LTS (Focal Fossa) ubuntu, Version: 20.04.4, Kernel: 5.15.0-87-generic | XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0) | Processor: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz | Cores: 12 | Logical: 24 |---------------------------------------------------------------------------------------------| | Total Memory (MB): 31768 | Free Memory: 15125 | Total Page/Swap (MB): 2047 | Free Page/Swap: 530 |---------------------------------------------------------------------------------------------| [2.810s] [ext: omni.kit.pipapi-0.0.0] startup [2.812s] [ext: omni.kit.pip_archive-0.0.0] startup [2.812s] [ext: omni.pip.compute-1.2.0] startup [2.813s] [ext: omni.pip.torch-2_0_1-2.0.2] startup [2.845s] [ext: omni.pip.cloud-1.0.1] startup [2.847s] [ext: omni.isaac.core_archive-2.2.1] startup [2.847s] [ext: omni.kit.loop-isaac-1.0.2] startup [2.849s] [ext: omni.kit.test-0.0.0] startup [2.851s] [ext: omni.kit.telemetry-0.4.0] startup [2.902s] [ext: omni.isaac.ml_archive-1.1.3] startup [2.904s] [ext: omni.mtlx-0.1.0] startup [2.906s] [ext: omni.usd.config-1.0.3] startup [2.919s] [ext: omni.gpucompute.plugins-0.0.0] startup [2.922s] [ext: omni.usd.libs-1.0.0] startup [3.180s] [ext: omni.appwindow-1.1.5] startup [3.496s] [ext: omni.kit.renderer.core-0.0.0] startup [4.599s] [ext: omni.kit.renderer.capture-0.0.0] starWarning: in Tf_PyLoadScriptModule at line 123 of /buildAgent/work/ac88d7d902b57417/USD/pxr/base/tf/pyUtils.cpp -- Import failed for module 'pxr.UsdSkel'! tup [4.606s] [ext: omni.kit.renderer.imgui-0.0.0] startup [4.823s] [ext: carb.audio-0.1.0] startup [4.852s] [ext: omni.ui-2.17.3] startup [4.886s] [ext: omni.uiaudio-1.0.0] startup [4.889s] [ext: omni.kit.mainwindow-1.0.0] startup [4.893s] [ext: omni.kit.uiapp-0.0.0] startup [4.893s] [ext: omni.usd.schema.anim-0.0.0] startup Traceback (most recent call last): File "/isaac-sim/kit/exts/omni.usd.libs/pxr/UsdSkel/__init__.py", line 25, inDo you have any ideas where this might come from? Thank you!