Closed cedricgoubard closed 7 months ago
It looks like the map failed to load from your stack trace:
E1114 09:35:34.744215 29 mapper.cpp:238] Failed to load map from file: /curobo/curobo/src/curobo/content/assets/scene/nvblox/srl_ur10_bins.nvblx
Can you check if this file exists in your clone of curobo: https://github.com/NVlabs/curobo/blob/main/src/curobo/content/assets/scene/nvblox/srl_ur10_bins.nvblx
Make sure this file is around 80mb and not just a lfs pointer. If it's a lfs pointer, you need to do the following:
sudo apt install git-lfs
cd curobo && git lfs pull
Hi, You were right, that solved the problem in IsaacSim... thank you! However, I am not loading any map in Python, but I am still getting the error. Here are some more details:
world_cfg = WorldConfig.from_dict(
{
"blox": {
"world": {
"pose": [0, 0, 0, 1, 0, 0, 0],
"integrator_type": "occupancy",
"voxel_size": 0.03,
}
}
}
)
motion_gen_config = MotionGenConfig.load_from_robot_config(
robot_cfg,
world_cfg,
tensor_args,
trajopt_tsteps=32,
collision_checker_type=CollisionCheckerType.BLOX,
use_cuda_graph=True,
num_trajopt_seeds=12,
num_graph_seeds=12,
interpolation_dt=0.03,
collision_activation_distance=0.025,
acceleration_scale=1.0,
self_collision_check=True,
maximum_trajectory_dt=0.25,
finetune_dt_scale=1.05,
fixed_iters_trajopt=True,
finetune_trajopt_iters=300,
minimize_jerk=True,
)
motion_gen = MotionGen(motion_gen_config)
motion_gen.warmup(warmup_js_trajopt=False)
Can you try calling motion_gen.world_collision.update_blox_hashes()
before calling motion_gen.warmup()
?
If that doesn't work, can you add a table obstacle similar to https://github.com/NVlabs/curobo/blob/e0e804e9062da33c13aa262a1da253d364b8768a/examples/isaac_sim/realsense_reacher.py#L186 ? This might help narrow down the issue you are having.
Thank you for the feedback; I did both, but unfortunately it looks like it did not change anything.
The values between the ==DEBUG==
flags in the stacktrace are prints I added to see the input of plan_single
.
Does pytest tests/nvblox_test.py
pass?
Hi,
The tests do not pass. I have been looking into it, but I am worried I might have made things worse: I am getting some ImportError
now, with nvblox_torch
not being properly installed.
More specifically:
Relevant tests and outputs:
#### Environment ####
python -V
Python 3.8.10
python -m pip -V
pip 23.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
pwd
/
#### CuRobo ####
python -m pip freeze | grep curobo
-e git+https://github.com/NVlabs/curobo.git@e0e804e9062da33c13aa262a1da253d364b8768a#egg=nvidia_curobo
python -c 'import curobo'
# No error
#### NVBlox Torch ####
python -m pip freeze | grep nvblox
# nothing
python -c 'import nvblox_torch'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'nvblox_torch
#### But it works if I am in the folder ####
cd curobo/nvblox_torch/
python -c 'import nvblox_torch'
# No error
#### Also, if I remove the '-e' from nvblox_torch/install.sh, it works
vim install.sh
sh ./install.sh
cd /
python -c 'import nvblox_torch'
# No error
Any idea what is causing this? I might have missed something in the nvblox_torch
install steps, which would explain the CUDA errors I was getting.
I am looking at refining the nvblox installation. Is your dockerfile starting from a pytorch container? If you can give me your dockerfile, I can use that to test the nvblox installation and possibly update it to be more robust.
Hi @balakumar-s Sorry about the delay, I was away for the last week.
I see you added the fixed_in_next_release
tag; should I wait for this release? Any idea when it'll be out?
Anyway, thanks again for your help :pray:
The release will be out by this Wednesday.
Thanks for the dockerfile. I will check to make sure the fix I have works in your docker config as well.
We just pushed a new release that should fix this error. Check this dockerfile for an example on how to compile nvblox: https://github.com/NVlabs/curobo/blob/58958bbcce4f7d8549268040237103ed904db8e4/docker/x86.dockerfile#L103
Closing this due to inactivity. Feel free to open this issue again if the latest release didn't fix it.
Hi @balakumar-s, Happy new year!
I've taken a look at the new installation instructions; I've managed to build everything in a container without the CXX11_ABI
(I'm using a pip-installed pytorch).
However, I'm still facing the same issue: the nvblox_torch
installation seems to succeed, but I then get an ModuleNotFoundError
.
The error:
python -c 'from nvblox_torch.mapper import Mapper'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'nvblox_torch'
I've considered using your container as a base image, but I need to install ROS noetic on top, and it does not officially support Ubuntu 23. I'll try to find a workaround if this does not work out, but any advice would be greatly appreciated :pray:.
Similar issue I have while executing motion_gen_reacher_nvblox.py
.
My setup is
Starts from docker isaac-sim:2023.1.0 python 3.10.13 torch 2.0.1+cu118 cuda 11.7 gpu 3090ti ubuntu 20.04
I installed nvblox with your guide.
There was no problem on motion_gen_reacher.py
before and after installation nvblox, but motion_gen_reacher_nvblox.py
gets problem after starting simulation.
I think warming up stage does not get problem, but motion generation makes cuda memory error.
error messege:
2024-01-10 07:08:25 [46,145ms] [Warning] [omni.physx.plugin] The rigid body at /World/panda/base_link has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for.
2024-01-10 07:08:25 [46,145ms] [Warning] [omni.physx.plugin] The rigid body at /World/panda/ee_link has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for.
2024-01-10 07:08:25 [46,302ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-01-10 07:08:25 [46,302ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-01-10 07:08:25 [46,357ms] [Warning] [omni.physx.plugin] The rigid body at /World/panda/base_link has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for.
2024-01-10 07:08:25 [46,357ms] [Warning] [omni.physx.plugin] The rigid body at /World/panda/ee_link has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for.
2024-01-10 07:08:32 [53,513ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-01-10 07:08:32 [53,513ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-01-10 07:08:32 [53,517ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-01-10 07:08:32 [53,517ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-01-10 07:08:32 [53,519ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-01-10 07:08:32 [53,519ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-01-10 07:08:32 [53,662ms] [Warning] [carb] Client omni.ui has acquired [omni::kit::renderer::IRenderer v1.9] 100 times. Consider accessing this interface with carb::getCachedInterface() (Performance warning)
2024-01-10 07:08:32 [53,662ms] [Warning] [carb] Client omni.ui has acquired [carb::svg::Svg v0.1] 100 times. Consider accessing this interface with carb::getCachedInterface() (Performance warning)
Traceback (most recent call last):
File "/isaac-sim/curobo/examples/isaac_sim/motion_gen_reacher_nvblox.py", line 294, in <module>
main()
File "/isaac-sim/curobo/examples/isaac_sim/motion_gen_reacher_nvblox.py", line 249, in main
result = motion_gen.plan_single(cu_js.unsqueeze(0), ik_goal, plan_config)
File "/isaac-sim/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1224, in plan_single
result = self._plan_attempts(
File "/isaac-sim/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1043, in _plan_attempts
result = self._plan_from_solve_state(
File "/isaac-sim/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1523, in _plan_from_solve_state
traj_result = self._solve_trajopt_from_solve_state(
File "/isaac-sim/kit/python/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/isaac-sim/curobo/src/curobo/wrap/reacher/motion_gen.py", line 912, in _solve_trajopt_from_solve_state
traj_result = trajopt_instance.solve_any(
File "/isaac-sim/curobo/src/curobo/wrap/reacher/trajopt.py", line 492, in solve_any
return self.solve_single(
File "/isaac-sim/curobo/src/curobo/wrap/reacher/trajopt.py", line 613, in solve_single
return self.solve_from_solve_state(
File "/isaac-sim/curobo/src/curobo/wrap/reacher/trajopt.py", line 579, in solve_from_solve_state
result = self.solver.solve(goal_buffer, seed_traj)
File "/isaac-sim/curobo/src/curobo/wrap/wrap_base.py", line 141, in solve
act_seq = self.optimize(seed, shift_steps=0)
File "/isaac-sim/curobo/src/curobo/wrap/wrap_base.py", line 71, in optimize
act_seq = opt.optimize(act_seq, shift_steps)
File "/isaac-sim/curobo/src/curobo/opt/opt_base.py", line 98, in optimize
torch.cuda.synchronize()
File "/isaac-sim/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.lx64/torch-2-0-1/torch/cuda/__init__.py", line 688, in synchronize
return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2024-01-10 07:08:40 [61,299ms] [Error] [carb.livestream.plugin] nvstPushStreamData error for eye 0, stream 0x7f3714018220: 0x800b0000
2024-01-10 07:08:40 [61,303ms] [Warning] [omni.core.ITypeFactory] Module /isaac-sim/kit/exts/omni.graph.action/bin/libomni.graph.action.plugin.so remained loaded after unload request.
2024-01-10 07:08:40 [61,307ms] [Warning] [carb] [Plugin: omni.spectree.delegate.plugin] Module /isaac-sim/kit/exts/omni.usd_resolver/bin/libomni.spectree.delegate.plugin.so remained loaded after unload request
2024-01-10 07:08:40 [61,311ms] [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,312ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,313ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,313ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-01-10 07:08:40 [61,316ms] [Warning] [carb.audio.context] 1 contexts were leaked
2024-01-10 07:08:40 [61,392ms] [Warning] [carb] Recursive unloadAllPlugins() detected!
2024-01-10 07:08:40 [61,403ms] [Warning] [omni.core.ITypeFactory] Module /isaac-sim/kit/exts/omni.activity.core/bin/libomni.activity.core.plugin.so remained loaded after unload request.
Is there any advice that I can try?
@Sepirence Can you create a new issue to track your error? We pushed a fix to nvblox_torch which might fix your error. Can you pull the latest commit from nvblox_torch repo?
@cedricgoubard I was able to fix the issue. You docker was unable to find python packages when installed in developer mode with pip install -e .
which we use for installing nvblox_torch.
I made changes to nvblox_torch repo that allows for pip install .
to work.
Here is the dockerfile with modifications:
Hi @balakumar-s,
Your fix mostly worked, thank you! I just had to switch back to editable mode, otherwise libpy_nvblox.so
was not found in /usr/local/lib/python3.8/dist-packages/nvblox_torch/cpp/build/
when actually using the package.
I can load everything properly, I'll test connecting a depth camera next week.
Thank you again for your help!
Hello,
I am getting an error when trying to use NVBlox with cuRobo. This happens both when I use the
motion_gen_reacher_nvblox.py
script with IsaacSim, and when I simply instantiate and warmup theMotionGen
class in Python. Everything is working properly if I usemotion_gen_reacher.py
, but notmotion_gen_reacher_nvblox.py
.The error is
RuntimeError: CUDA error: an illegal memory access was encountered
.My setup is:
Stacktrace (isaacsim)
``` /isaac-sim/python.sh curobo/examples/isaac_sim/motion_gen_reacher_nvblox.py --robot /ros_ws/src/curobo_ros/cfg/amiga.yaml --visualize_spheres [14.954s] app ready [15.212s] RTX ready [15.356s] Simulation App Startup Complete torchtyping could not be imported, falling back to basic types 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amiga_arm_base_link-base_link_inertia is not a valid usd path, modifying to amiga_arm_base_link_base_link_inertia 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amiga_arm_base_link-base_fixed_joint is not a valid usd path, modifying to amiga_arm_base_link_base_fixed_joint 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amiga_arm_wrist_3-flange is not a valid usd path, modifying to amiga_arm_wrist_3_flange 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amiga_arm_flange-tool0 is not a valid usd path, modifying to amiga_arm_flange_tool0 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amig_arm_wrist_3_link-amiga_grip_camera_mount_link is not a valid usd path, modifying to amig_arm_wrist_3_link_amiga_grip_camera_mount_link /World/amiga/Looks/material_Material_001_1 WARNING: Logging before InitGoogleLogging() is written to STDERR I1114 09:35:34.738322 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox15VoxelBlockLayerINS_9TsdfVoxelEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. I1114 09:35:34.738354 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox15VoxelBlockLayerINS_10ColorVoxelEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. I1114 09:35:34.738360 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox15VoxelBlockLayerINS_14OccupancyVoxelEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. I1114 09:35:34.738365 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox15VoxelBlockLayerINS_9EsdfVoxelEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. I1114 09:35:34.738370 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox10BlockLayerINS_9MeshBlockEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. E1114 09:35:34.744175 29 sqlite_database.cpp:192] Preparing query failed: file is not a database E1114 09:35:34.744215 29 mapper.cpp:238] Failed to load map from file: /curobo/curobo/src/curobo/content/assets/scene/nvblox/srl_ur10_bins.nvblx warming up... /curobo/curobo/src/curobo/opt/particle/parallel_mppi.py:224: UserWarning: FALLBACK path has been taken inside: runCudaFusionGroup. This is an indication that codegen Failed for some reason. To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback` (Triggered internally at ../torch/csrc/jit/codegen/cuda/manager.cpp:331.) cov_update = jit_diag_a_cov_update(w, actions, self.mean_action) /curobo/curobo/src/curobo/opt/particle/parallel_mppi.py:297: UserWarning: FALLBACK path has been taken inside: runCudaFusionGroup. This is an indication that codegen Failed for some reason. To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback` (Triggered internally at ../torch/csrc/jit/codegen/cuda/manager.cpp:331.) new_cov = jit_blend_cov(self.cov_action, cov_update, self.step_size_cov, self.kappa) /curobo/curobo/src/curobo/opt/particle/parallel_mppi.py:301: UserWarning: FALLBACK path has been taken inside: runCudaFusionGroup. This is an indication that codegen Failed for some reason. To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback` (Triggered internally at ../torch/csrc/jit/codegen/cuda/manager.cpp:331.) new_mean = jit_blend_mean(self.mean_action, new_mean, self.step_size_mean) Traceback (most recent call last): File "curobo/examples/isaac_sim/motion_gen_reacher_nvblox.py", line 300, inHere is the code I tried when using Python only:
I added some debug prints in
curobo/wrap/reacher/motion_gen.py
; here is whatplan_single
is getting as inputs:Here are my installation steps, in case it helps
```dockerfile ################################################################################################### ######################################### CMake version ########################################### ################################################################################################### RUN wget https://cmake.org/files/v3.19/cmake-3.19.5.tar.gz && tar -xvzf cmake-3.19.5.tar.gz RUN DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential checkinstall zlib1g-dev libssl-dev RUN cd cmake-3.19.5 && ./bootstrap RUN cd cmake-3.19.5 && make -j `nproc` && make install RUN rm -rf cmake-3.19.5.tar.gz #################################################################################################### ############################################## nvblox ############################################## #################################################################################################### RUN git clone https://github.com/valtsblukis/nvblox.git RUN apt-get install -y libgoogle-glog-dev libgtest-dev libgflags-dev python3-dev libsqlite3-dev RUN cd /usr/src/googletest && sudo cmake . && sudo cmake --build . --target install RUN cd nvblox && cd nvblox && mkdir build && cd build && \ cmake .. -DPRE_CXX11_ABI_LINKABLE=ON -DCMAKE_CUDA_FLAGS="-gencode=arch=compute_86,code=sm_86" -DCMAKE_CUDA_ARCHITECTURES="86" -DSTDGPU_CUDA_ARCHITECTURE_FLAGS_USER="86" -DBUILD_FOR_ALL_ARCHS=ON &&\ make -j `nproc` &&\ make install RUN git clone https://github.com/nvlabs/nvblox_torch.git RUN cd nvblox_torch && sh install.sh #################################################################################################### ############################################## curobo ############################################## #################################################################################################### RUN git clone https://github.com/NVlabs/curobo.git RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections ENV PATH="${PATH}:/opt/hpcx/ompi/bin" ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/hpcx/ompi/lib" ENV LD_LIBRARY_PATH="/usr/local/lib:${LD_LIBRARY_PATH}" RUN cd curobo && pip install -e . --no-build-isolation ```Do you have any ideas what may be causing it? Thank you for your help!