Error when using isaacgym

[Norweig1an]

Hi, thank you for your awesome work! When we tried to use the project for retargeting in the issacgym environment, we encountered an error: TypeError: No Python class registered for C++ class std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > The error seemed to be caused by a conflict between isaacgym and pinocchio, where if we read the model from urdf and run print(model.names) after import isaacgym the above error occurs.

Would you please give us some ideas about how to deal with it? Our isaacgym version is 1.0rc4.

[yzqin]

Could you please provide a detailed script to reproduce this error? I am using isaacgym myself and I do not encounter this error. Here is the screenshot of my terminal command.

Python 3.8.19 (default, Mar 20 2024, 19:58:24) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dex_retargeting
>>> import isaacgym
Importing module 'gym_38' (/home/yuzhe/miniconda3/envs/debug/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/yuzhe/miniconda3/envs/debug/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json

Or with a different import order:

Python 3.8.19 (default, Mar 20 2024, 19:58:24) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import isaacgym
Importing module 'gym_38' (/home/yuzhe/miniconda3/envs/debug/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/yuzhe/miniconda3/envs/debug/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
>>> import dex_retargeting

Here is my version of pin and isaacgym

pip3 list | grep isaacgym
isaacgym                 1.0rc4

pip3 list | grep pin
pin                      2.7.0

[y8han]

Hi, I encountered the same issue here, and I feel this could be some conflicts between isaacgym simulator and dex-retargeting codebase. For verification, I just added one line import isaacgym to the python script detect_from_video.py and run the demo command

python3 detect_from_video.py \
  --robot-name allegro \
  --video-path data/human_hand_video.mp4 \
  --retargeting-type dexpilot \
  --hand-type right \
  --output-path data/allegro_joints.pkl 

Then, I got the same error message TypeError: No Python class registered for C++ class std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > Without importing the isaacgym, this script works fine as expected.

[yzqin]

Thank you for reporting this issue. I've been able to reproduce the bugs on my machine following @y8han. The problem appears to stem from isaacgym overriding the built-in pybind11 registrar of C++ std::vector<std::string>. This affects all pybind11-based Python libraries that explicitly return a std::vector, .e.g pinocchio.

A temporary workaround involves adjusting the import order to ensure Pinocchio loads before isaacgym. Additionally, since isaacgym needs to be loaded before torch, the recommended import order is:

pinocchio -> isaacgym -> torch

Here's an example of how to structure your imports:

# Other unrelated libraries not depending on pybind11
import other_unrelated_libs

import pinocchio
import isaacgym
import torch
import dex_retargeting

# More unrelated libraries
import more_unrelated_libs

To fix the error in detect_from_video.py mentioned by @y8han, you can modify the import section as follows:

import pickle
from pathlib import Path

import cv2
import tqdm
import tyro

import pinocchio
import isaacgym
import torch
from dex_retargeting.constants import RobotName, RetargetingType, HandType, get_default_config_path
from dex_retargeting.retargeting_config import RetargetingConfig
from dex_retargeting.seq_retarget import SeqRetargeting
from single_hand_detector import SingleHandDetector

This solution should resolve the immediate issue. However, if anyone discovers a more elegant solution, please feel free to discuss it in this issue thread.

[y8han]

Thank you for reporting this issue. I've been able to reproduce the bugs on my machine following @y8han. The problem appears to stem from isaacgym overriding the built-in pybind11 registrar of C++ std::vector<std::string>. This affects all pybind11-based Python libraries that explicitly return a std::vectorstd::string, .e.g pinocchio.

A temporary workaround involves adjusting the import order to ensure Pinocchio loads before isaacgym. Additionally, since isaacgym needs to be loaded before torch, the recommended import order is:

pinocchio -> isaacgym -> torch

Here's an example of how to structure your imports:

# Other unrelated libraries not depending on pybind11
import other_unrelated_libs

import pinocchio
import isaacgym
import torch
import dex_retargeting

# More unrelated libraries
import more_unrelated_libs

To fix the error in detect_from_video.py mentioned by @y8han, you can modify the import section as follows:

import pickle
from pathlib import Path

import cv2
import tqdm
import tyro

import pinocchio
import isaacgym
import torch
from dex_retargeting.constants import RobotName, RetargetingType, HandType, get_default_config_path
from dex_retargeting.retargeting_config import RetargetingConfig
from dex_retargeting.seq_retarget import SeqRetargeting
from single_hand_detector import SingleHandDetector

This solution should resolve the immediate issue. However, if anyone discovers a more elegant solution, please feel free to discuss it in this issue thread.

This works perfectly on my local machine! Thank you for your quick reply!