[Question] Importing custom ROS messages to use with Orbit

haastregt commented 10 months ago

Question

I want to use a custom ROS message to control my robot in Orbit. How can I do this without breaking the Isaac Sim interface?

What I tried

In order to use a custom message with Isaac Sim, I followed this tutorial from their documentation: https://docs.omniverse.nvidia.com/isaacsim/latest/ros_tutorials/tutorial_ros_custom_message.html

This works fine, however if I want to launch my python script with orbit, I can't manage to import the message without problems launching the isaac sim. I tried the following:

Sourcing my ROS workspace and running my python script outside of the conda environment will fail because it cant find the orbit modules (understandable)
Sourcing my ROS workspace outside of the conda environment and running the python script in the conda environment with orbit -p my_script.py will launch isaac sim but give a ModuleNotFound error on my custom message
Sourcing my ROS workspace inside the conda environment and running the python script from there will fail starting isaac sim with many errors similar to: 2023-12-08 14:59:44 [6,331ms] [Error] [omni.ext.plugin] [ext: omni.isaac.quadruped-1.2.2] Failed to startup python extension. 2023-12-08 14:59:44 [6,427ms] [Error] [omni.ext.python] NameError: name '_C' is not defined

I have to say I do not have a great understanding how conda environments work and how this is connected with the custom ROS implementation in Isaac Sim.

I hope I was able to clearly explain my problem. What would be the correct way to source the relevant workspaces or alternatively is there a workaround how I could import these custom messages? Thanks!

pascal-roth commented 10 months ago

Hi,

can you try to add source /opt/ros/noetic/setup.bash (or any other ros version you are using) in _isaac_sim/setup_python_env.sh script. Then you should be able to find your custom ROS messages within orbit. I tested it without the conda environment, i.e., directly call ./orbit.sh -p your_script.py without sourcing conda first. Possible, that you have to run ./orbit.sh -i again.

haastregt commented 10 months ago

Hi Pascal, thank you for taking your time to help me out!

I tried this within the conda environment, both with sourcing /opt/ros/noetic/setup.bash and the workspace where I have my custom messages. Unfortunately it didn't work here, I still get module not found errors on my custom message import.

When I want to try to run ./orbit.sh -p my_script.py outside the conda environment, I get the same errors as before (Failed to startup python extensions and name '_C' is not defined) as well as import errors. Probably because orbit is not installed outside of conda. However when I try to run ./orbit.sh -i outside my conda environment, I get the following error:

[INFO] Installing extensions inside orbit repository...                                                                                                                                                   
     module: /home/teleop/Orbit/source/extensions/omni.isaac.orbit_envs
Obtaining file:///home/teleop/Orbit/source/extensions/omni.isaac.orbit_envs
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [28 lines of output]
      Traceback (most recent call last):
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/importlib_metadata/_compat.py", line 9, in <module>
          from typing import Protocol
      ImportError: cannot import name 'Protocol' from 'typing' (/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/typing.py)

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "<string>", line 36, in <module>
        File "<pip-setuptools-caller>", line 14, in <module>
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.pip_archive/pip_prebundle/setuptools/__init__.py", line 18, in <module>
          from setuptools.dist import Distribution
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.pip_archive/pip_prebundle/setuptools/dist.py", line 34, in <module>
          from ._importlib import metadata
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.pip_archive/pip_prebundle/setuptools/_importlib.py", line 39, in <module>
          disable_importlib_metadata_finder(metadata)
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.pip_archive/pip_prebundle/setuptools/_importlib.py", line 12, in disable_importlib_metadata_finder
          import importlib_metadata
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 17, in <module>
          from . import _adapters, _meta, _py39compat
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/importlib_metadata/_meta.py", line 1, in <module>
          from ._compat import Protocol
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/importlib_metadata/_compat.py", line 12, in <module>
          from typing_extensions import Protocol  # type: ignore
        File "/opt/ros/noetic/lib/python3/dist-packages/typing_extensions.py", line 874
          def TypedDict(typename, fields=_marker, /, *, total=True, **kwargs):
                                                  ^
      SyntaxError: invalid syntax
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
There was an error running python
     module: /home/teleop/Orbit/source/extensions/omni.isaac.orbit

Are there any installation steps I missed? The installation guide only shows how to install Orbit using a conda environment. The approach I took trying to install orbit outside of conda is to just omit the steps setting up a conda environment.

I also tried running ./orbit.sh -p "import omni.isaac.orbit; print('Orbit configuration is now complete.')" outside of any conda environment, which results in:

[INFO] Using python from: /home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/python.sh                                                                                                                   
/home/teleop/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/bin/python3: can't open file 'import': [Errno 2] No such file or directory
There was an error running python

pascal-roth commented 10 months ago

I haven't experienced any such install errors before. My best guess would be your Isaac Sim version. You are still using isaac_sim-2022.2.0, which has a lot of issues. Could you try it again with the current 2023-1-... version?

haastregt commented 10 months ago

I had to downgrade my Isaac Sim version because of the following issue: https://github.com/NVIDIA-Omniverse/Orbit/issues/151

Nonetheless I can try version 2023.1.0 again and see if I can just work around the issues with that version.

pascal-roth commented 10 months ago

I cannot reproduce this issue on my side (working with 2023.1.0).

Please let me know if that fixes anything for you.

haastregt commented 10 months ago

Apparently, the install error is caused because of the 'source /opt/ros/noetic/setup.bash' I added in _isaac_sim/setup_python_env.sh. I removed this line and ran ./orbit.sh -i again, now it installed.

However it does not fix the original issue of not being able to import the custom messages, once again giving module not found errors when not sourcing and Failed to startup python extensions and name '_C' is not defined errors when sourcing.

I will let you know how it goes with Isaac Sim 2023.1.0

haastregt commented 10 months ago

I tried with Isaac Sim 2023.1.0-hotfix.1 (I dropped the conda environment). After installation, I still got a module not found error on my custom messages when running straight from the terminal with 'orbit -p my_script.py'. This also happened when I sourced my ros version and/or workspace in the terminal I ran this command from. In that way it at least seems more robust.

When sourcing ros and/or my workspace in _isaac_sim/setup_python_env.sh, and running orbit -i, I get the following error:

[INFO] Installing extensions inside orbit repository...                                                                                                                                                   
     module: /home/teleop/Orbit/source/extensions/omni.isaac.orbit_envs
Obtaining file:///home/teleop/Orbit/source/extensions/omni.isaac.orbit_envs
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error

  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      Traceback (most recent call last):
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/teleop/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
          return hook(config_settings)
        File "/tmp/pip-build-env-08pcak4a/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 441, in get_requires_for_build_editable
          return self.get_requires_for_build_wheel(config_settings)
        File "/tmp/pip-build-env-08pcak4a/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-08pcak4a/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-08pcak4a/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 480, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-08pcak4a/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 10, in <module>
      ModuleNotFoundError: No module named 'toml'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
There was an error running python
     module: /home/teleop/Orbit/source/extensions/omni.isaac.orbit

This error now maintains even when I remove the source lines from _isaac_sim/setup_python_env.sh and try orbit -i. Are there some kind of old configuration files I have to clean up?

Other things I tried:

pip install toml (Requirement already satisfied: toml in /home/teleop/.local/lib/python3.8/site-packages (0.10.2))
${ISAACSIM_PATH}/isaac-sim.sh --reset-user

python -c "import omni.isaac.orbit; print('Orbit configuration is now complete.')" returns:

Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'omni'

orbit -p "import omni,isaac.orbit; print('succes')" returns

[INFO] Using python from: /home/teleop/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/python.sh                                                                                                          
/home/teleop/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/python/bin/python3: can't open file '/home/teleop/Orbit/import': [Errno 2] No such file or directory
There was an error running python

Mayankm96 commented 10 months ago

Hopefully, the merge happens soon, but you can start using the devel branch. It should be working with 2023.1 version of Isaac Sim.

haastregt commented 10 months ago

Updating to 2023.1.0 actually did solve the issue and was necessary to make it work.

The second part of the problem was a breaking update over the weekend to the software I wanted to import the message from.

Sorry for not noticing that sooner, thank you very much for the help troubleshooting!

isaac-sim / IsaacLab

[Question] Importing custom ROS messages to use with Orbit #164

Question

What I tried