balakumar-s
commented
9 months ago It looks like the map failed to load from your stack trace:
E1114 09:35:34.744215 29 mapper.cpp:238] Failed to load map from file: /curobo/curobo/src/curobo/content/assets/scene/nvblox/srl_ur10_bins.nvblxCan you check if this file exists in your clone of curobo: https://github.com/NVlabs/curobo/blob/main/src/curobo/content/assets/scene/nvblox/srl_ur10_bins.nvblx
Make sure this file is around 80mb and not just a lfs pointer. If it's a lfs pointer, you need to do the following:
sudo apt install git-lfs
cd curobo && git lfs pull
cedricgoubard
Sepirence
Hello,
I am getting an error when trying to use NVBlox with cuRobo. This happens both when I use the
motion_gen_reacher_nvblox.pyscript with IsaacSim, and when I simply instantiate and warmup theMotionGenclass in Python. Everything is working properly if I usemotion_gen_reacher.py, but notmotion_gen_reacher_nvblox.py.The error is
RuntimeError: CUDA error: an illegal memory access was encountered.My setup is:
Stacktrace (isaacsim)
``` /isaac-sim/python.sh curobo/examples/isaac_sim/motion_gen_reacher_nvblox.py --robot /ros_ws/src/curobo_ros/cfg/amiga.yaml --visualize_spheres [14.954s] app ready [15.212s] RTX ready [15.356s] Simulation App Startup Complete torchtyping could not be imported, falling back to basic types 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amiga_arm_base_link-base_link_inertia is not a valid usd path, modifying to amiga_arm_base_link_base_link_inertia 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amiga_arm_base_link-base_fixed_joint is not a valid usd path, modifying to amiga_arm_base_link_base_fixed_joint 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amiga_arm_wrist_3-flange is not a valid usd path, modifying to amiga_arm_wrist_3_flange 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amiga_arm_flange-tool0 is not a valid usd path, modifying to amiga_arm_flange_tool0 2023-11-14 09:34:46 [19,734ms] [Warning] [omni.isaac.urdf] The path amig_arm_wrist_3_link-amiga_grip_camera_mount_link is not a valid usd path, modifying to amig_arm_wrist_3_link_amiga_grip_camera_mount_link /World/amiga/Looks/material_Material_001_1 WARNING: Logging before InitGoogleLogging() is written to STDERR I1114 09:35:34.738322 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox15VoxelBlockLayerINS_9TsdfVoxelEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. I1114 09:35:34.738354 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox15VoxelBlockLayerINS_10ColorVoxelEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. I1114 09:35:34.738360 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox15VoxelBlockLayerINS_14OccupancyVoxelEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. I1114 09:35:34.738365 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox15VoxelBlockLayerINS_9EsdfVoxelEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. I1114 09:35:34.738370 29 layer_cake_impl.h:32] Adding Layer with type: N6nvblox10BlockLayerINS_9MeshBlockEEE, voxel_size: 0.03, and memory_type: kDevice to LayerCake. E1114 09:35:34.744175 29 sqlite_database.cpp:192] Preparing query failed: file is not a database E1114 09:35:34.744215 29 mapper.cpp:238] Failed to load map from file: /curobo/curobo/src/curobo/content/assets/scene/nvblox/srl_ur10_bins.nvblx warming up... /curobo/curobo/src/curobo/opt/particle/parallel_mppi.py:224: UserWarning: FALLBACK path has been taken inside: runCudaFusionGroup. This is an indication that codegen Failed for some reason. To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback` (Triggered internally at ../torch/csrc/jit/codegen/cuda/manager.cpp:331.) cov_update = jit_diag_a_cov_update(w, actions, self.mean_action) /curobo/curobo/src/curobo/opt/particle/parallel_mppi.py:297: UserWarning: FALLBACK path has been taken inside: runCudaFusionGroup. This is an indication that codegen Failed for some reason. To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback` (Triggered internally at ../torch/csrc/jit/codegen/cuda/manager.cpp:331.) new_cov = jit_blend_cov(self.cov_action, cov_update, self.step_size_cov, self.kappa) /curobo/curobo/src/curobo/opt/particle/parallel_mppi.py:301: UserWarning: FALLBACK path has been taken inside: runCudaFusionGroup. This is an indication that codegen Failed for some reason. To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback` (Triggered internally at ../torch/csrc/jit/codegen/cuda/manager.cpp:331.) new_mean = jit_blend_mean(self.mean_action, new_mean, self.step_size_mean) Traceback (most recent call last): File "curobo/examples/isaac_sim/motion_gen_reacher_nvblox.py", line 300, inHere is the code I tried when using Python only:
I added some debug prints in
curobo/wrap/reacher/motion_gen.py; here is whatplan_singleis getting as inputs:Here are my installation steps, in case it helps
```dockerfile ################################################################################################### ######################################### CMake version ########################################### ################################################################################################### RUN wget https://cmake.org/files/v3.19/cmake-3.19.5.tar.gz && tar -xvzf cmake-3.19.5.tar.gz RUN DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential checkinstall zlib1g-dev libssl-dev RUN cd cmake-3.19.5 && ./bootstrap RUN cd cmake-3.19.5 && make -j `nproc` && make install RUN rm -rf cmake-3.19.5.tar.gz #################################################################################################### ############################################## nvblox ############################################## #################################################################################################### RUN git clone https://github.com/valtsblukis/nvblox.git RUN apt-get install -y libgoogle-glog-dev libgtest-dev libgflags-dev python3-dev libsqlite3-dev RUN cd /usr/src/googletest && sudo cmake . && sudo cmake --build . --target install RUN cd nvblox && cd nvblox && mkdir build && cd build && \ cmake .. -DPRE_CXX11_ABI_LINKABLE=ON -DCMAKE_CUDA_FLAGS="-gencode=arch=compute_86,code=sm_86" -DCMAKE_CUDA_ARCHITECTURES="86" -DSTDGPU_CUDA_ARCHITECTURE_FLAGS_USER="86" -DBUILD_FOR_ALL_ARCHS=ON &&\ make -j `nproc` &&\ make install RUN git clone https://github.com/nvlabs/nvblox_torch.git RUN cd nvblox_torch && sh install.sh #################################################################################################### ############################################## curobo ############################################## #################################################################################################### RUN git clone https://github.com/NVlabs/curobo.git RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections ENV PATH="${PATH}:/opt/hpcx/ompi/bin" ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/hpcx/ompi/lib" ENV LD_LIBRARY_PATH="/usr/local/lib:${LD_LIBRARY_PATH}" RUN cd curobo && pip install -e . --no-build-isolation ```Do you have any ideas what may be causing it? Thank you for your help!