search

robotology / gym-ignition

Framework for developing OpenAI Gym robotics environments simulated with Ignition Gazebo
https://robotology.github.io/gym-ignition
GNU Lesser General Public License v3.0
220 stars 26 forks source link

Failing to install `scenario` on MacOS #422

Open lrapetti opened 2 years ago

lrapetti commented 2 years ago

Description:

I noticed that macOS is not officially supported. I thought it was good to report anyway.

Steps to reproduce

I tried to install it with

pip install scenario

Output can be found below

Additional context

Output 
$pip install scenario
Collecting scenario
  Downloading scenario-1.3.1.tar.gz (151 kB)
     |████████████████████████████████| 151 kB 352 kB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting packaging
  Using cached packaging-21.2-py3-none-any.whl (40 kB)
Collecting pyparsing<3,>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Building wheels for collected packages: scenario
  Building wheel for scenario (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/bin/python3.9 /Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/tmpb33hqy26
       cwd: /private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10
  Complete output (80 lines):
  running bdist_wheel
  running build
  running build_ext
  -- The C compiler identification is AppleClang 12.0.5.12050022
  -- The CXX compiler identification is AppleClang 12.0.5.12050022
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/local/opt/ccache/libexec/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/local/opt/ccache/libexec/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Looking for pthread.h
  -- Looking for pthread.h - found
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
  -- Found Threads: TRUE
  -- Found Python3: /Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/bin/python3.9 (found version "3.9.6") found components: Interpreter Development Development.Module Development.Embed
  -- Using Python: /Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/bin/python3.9
  CMake Error at CMakeLists.txt:205 (message):
    Found no Ignition distribution for PyPI package

  -- Configuring incomplete, errors occurred!
  See also "/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10/build/temp.macosx-11-x86_64-3.9_ScenarioCMakeProject/CMakeFiles/CMakeOutput.log".

  ==> Configuring:
  $ cmake -S /private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10 -B /private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10/build/temp.macosx-11-x86_64-3.9_ScenarioCMakeProject -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10/build/lib.macosx-11-x86_64-3.9/scenario -DSCENARIO_CALL_FROM_SETUP_PY:BOOL=ON -DSCENARIO_BUILD_SHARED_LIBRARY:BOOL=OFF -DPython3_EXECUTABLE:PATH=/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/bin/python3.9

  ==> Building:
  $ cmake --build /private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10/build/temp.macosx-11-x86_64-3.9_ScenarioCMakeProject --config Release

  ==> Installing:
  $ cmake --install /private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10/build/temp.macosx-11-x86_64-3.9_ScenarioCMakeProject

  Traceback (most recent call last):
    File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in 
      main()
    File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 204, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-build-env-r6jc4hrg/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 230, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-build-env-r6jc4hrg/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
      self.run_setup()
    File "/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-build-env-r6jc4hrg/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 10, in 
      setuptools.setup(
    File "/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-build-env-r6jc4hrg/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 159, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-build-env-r6jc4hrg/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-build-env-r6jc4hrg/overlay/lib/python3.9/site-packages/cmake_build_extension/build_extension.py", line 110, in run
      self.build_extension(ext)
    File "/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-build-env-r6jc4hrg/overlay/lib/python3.9/site-packages/cmake_build_extension/build_extension.py", line 224, in build_extension
      subprocess.check_call(configure_command)
    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '-S', '/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10', '-B', '/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10/build/temp.macosx-11-x86_64-3.9_ScenarioCMakeProject', '-GNinja', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX:PATH=/private/var/folders/v4/8y04pr2912dd1n1pryz743r80000gp/T/pip-install-d6fi17h8/scenario_93d92748aa99469aba71337237b49d10/build/lib.macosx-11-x86_64-3.9/scenario', '-DSCENARIO_CALL_FROM_SETUP_PY:BOOL=ON', '-DSCENARIO_BUILD_SHARED_LIBRARY:BOOL=OFF', '-DPython3_EXECUTABLE:PATH=/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/bin/python3.9']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for scenario
Failed to build scenario
ERROR: Could not build wheels for scenario which use PEP 517 and cannot be installed directly

Environment

diegoferigo commented 2 years ago

Hi @lrapetti, yes as you read from the support policy, macOS is not currently supported. If you inspect the CMake output, you'll notice that it fails with the following error:

  CMake Error at CMakeLists.txt:205 (message):
    Found no Ignition distribution for PyPI package

You should install Ignition Gazebo manually before attempting to install scenario. Open Robotics currently supports macOS even though I cannot guarantee that scenario will work ootb. If you are willing do iterate on possible errors that might emerge, I can provide support.

I actually already did some work on macOS in #406 while attempting to finalize https://github.com/conda-forge/staged-recipes/pull/16582, but the last update was https://github.com/conda-forge/staged-recipes/pull/16582#issuecomment-950125662 in which everything was building correctly on macOS (a nice start), but tests were segfaulting.

diegoferigo commented 2 years ago

I just remembered that we had a previous discussion regarding macOS in #100, not sure if things have changes since.

lrapetti commented 2 years ago

Open Robotics currently supports macOS even though I cannot guarantee that scenario will work ootb.

Following the ignition gazebo installation, I was able to install both scenario and gym-ignition via pip without errors.

lrapetti commented 2 years ago

Still, I am experiencing the command line tool mentioned in https://github.com/robotology/gym-ignition/issues/100#issuecomment-552233722. It looks like the problem is still present in the ignition version I have installed.

I was able to run some gym-ignition code such as gym_ignition_models, but not those using gazebo such as the examples:

$python3 launch_cartpole.py
Traceback (most recent call last):
  File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/scenario/__init__.py", line 141, in check_gazebo_installation
    result = subprocess.run(command, capture_output=True, text=True, check=True)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ign', 'gazebo', '--versions']' returned non-zero exit status 255.
diegoferigo commented 2 years ago

What fails here is just a check for a compatible Ignition Gazebo version, it's not strictly needed during runtime.

Can you please report if the following command executed from a normal terminal succeeds, and if it does, its output?

ign gazebo --versions

If it does not succeed, you can try commenting out the following line in /Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/scenario/__init__.py:

https://github.com/robotology/gym-ignition/blob/aafd7fb3b97f36fa460127b70f5968167eff8fce/scenario/bindings/__init__.py#L227

lrapetti commented 2 years ago

Can you please report if the following command executed from a normal terminal succeeds, and if it does, its output?

ign gazebo --versions

Since there is the problem with the command line tool, I am getting the following

ign gazebo --versions
I cannot find any available 'ign' command:
    * Did you install any ignition library?
    * Did you set the IGN_CONFIG_PATH environment variable?
        E.g.: export IGN_CONFIG_PATH=$HOME/local/share/ignition

So I commted out the lines as you where suggesting, but I am getting the following

 $python3 launch_cartpole.py
Traceback (most recent call last):
  File "/Users/lorenzorapetti/Software/dic-iit/gym-ignition/examples/python/launch_cartpole.py", line 9, in <module>
    from gym_ignition.utils import logger
  File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/gym_ignition/__init__.py", line 14, in <module>
    from gym_ignition.utils import resource_finder
  File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/gym_ignition/utils/__init__.py", line 5, in <module>
    from . import logger, math, misc, resource_finder, scenario
  File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/gym_ignition/utils/math.py", line 10, in <module>
    from scenario import gazebo as scenario
  File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/scenario/__init__.py", line 228, in <module>
    import_gazebo()
  File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/scenario/__init__.py", line 203, in import_gazebo
    import scenario.bindings.gazebo
  File "/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/scenario/bindings/gazebo.py", line 13, in <module>
    from . import _gazebo
ImportError: dlopen(/Users/lorenzorapetti/Software/dic-iit/ADAM/adamEnv/lib/python3.9/site-packages/scenario/bindings/_gazebo.so, 10): Library not loaded: /usr/local/opt/urdfdom/lib/liburdfdom_sensor.3.0.dylib
  Referenced from: /usr/local/opt/sdformat12/lib/libsdformat12.12.dylib
  Reason: image not found

I guess that is due to some conflict or wrong configuration on my laptop. Indeed I see the following

$ ls /usr/local/opt/urdfdom/lib
liburdfdom_model.1.0.dylib      liburdfdom_sensor.dylib
liburdfdom_model.dylib          liburdfdom_world.1.0.dylib
liburdfdom_model_state.1.0.dylib    liburdfdom_world.dylib
liburdfdom_model_state.dylib        pkgconfig
liburdfdom_sensor.1.0.dylib     urdfdom
diegoferigo commented 2 years ago

I see that you have liburdfdom_sensor.1.0.dylib while sdformat12 requires liburdfdom_sensor.3.0.dylib. Could it be a problem of the packages you already had installed in your system before installing Ignition Gazebo? Can you try to update the library?

Another option is to try using a conda environment instead of relying on the official macOS packages. Though, even with conda, something could be missing.

srmainwaring commented 2 years ago

Not sure if this is the right place for reporting experimental environments, but I have had success in getting ScenarIO running on macOS (with GUI) using development branches that add Metal support to Ignition (which will be slowly making there way into released versions - hopefully in time for Ignition Garden). I'd be happy to outline the approach here or somewhere suitable if there is an interest. This looks like a great tool btw! I expect to be using it a good deal.

diegoferigo commented 2 years ago

Hi @srmainwaring, this is great news, thanks! I'm not sure if @lrapetti has any news after his last attempts, in any case it would be very helpful if you can describe your working setup and whether there's anything missing on our side to run on macOS.

lrapetti commented 2 years ago

I'm not sure if @lrapetti has any news after his last attempts

Sorry, I didn't proceed further at this stage. In case I'll let you know when I start again this investigation.

srmainwaring commented 2 years ago

Hi @diegoferigo - here's the description, it may take a number of edits to get right as it is fairly involved at this stage. I hope it will simplify as more of the development work is merged into the relevant repos.

Install Ignition on macOS

The first step is to get the Ignition graphics API running properly on macOS. I've documented how to install Ignition with Metal rendering on macOS here: https://github.com/srmainwaring/ign-rendering/wiki. Over time this page should become redundant as more features are merged into the https://github.com/ignitionrobotics repos.

Install iDynTree

The installation on macOS largely follows the iDynTree install instructions with an extra step to install to /usr/local/Cellar/idyntree/1.0.0 and use brew to link into /usr/local.

Clone and install:

# clone to ~/robotology
$ cd ~/robotology
$ git clone https://github.com/robotology/idyntree.git
$ cd idyntree
$ mkdir build && cd build
$ cmake .. \
  -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/idyntree/1.0.0 \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DIDYNTREE_USES_PYTHON=ON
$ make -j16
$ make install
$ brew link idyntree

Check the installation:

>>> import idyntree
>>> import idyntree.bindings as iDynTree
>>> p = iDynTree.Position()
>>> print(p)
1.46135e+224 1.0947e+248 1.02265e-259

Run the example:

$ cd ~/robotology/idyntree/examples/python
$ python KinDynComputationsTutorial.py

Install ScenarIO

We use a patched version of ScenarIO which includes dependency definitions for Ignition Garden. The libraries are installed to /usr/local/Cellar/scenario/1.0.0 and then brew linked into /usr/local:

# clone to ~/robotology
$ cd ~/robotology
$ git clone https://github.com/robotology/gym-ignition.git
$ git remote add fork https://github.com/srmainwaring/gym-ignition.git
$ git checkout fork/feature/ign-garden-macos
$ cd gym-ignition/scenario
$ mkdir build && cd build
$ cmake .. \
  -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/scenario/1.0.0 \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DSCENARIO_USE_IGNITION=ON \
  -DSCENARIO_ENABLE_BINDINGS=ON
$ make -j16
$ make install
$ brew link scenario
$ cd ~/robotology/gym-ignition
$ pip install --user -e scenario
$ pip install --user -e .

I have not used a Python virtual environment for my installation, but that should also work.

Configuration

ScenarIO uses a custom physics plugin that overrides the defaults in World.cpp

https://github.com/robotology/gym-ignition/blob/fa14f7f8f667cf84bf226756cea158ec0f5c3d23/scenario/src/gazebo/src/World.cpp#L251-L284

It expects to find the system plugin libPhysicsSystem.dylib so we need to add the install location to IGN_GAZEBO_PLUGIN_INSTALL_DIR when running C++ scenarios:

export IGN_GAZEBO_PLUGIN_INSTALL_DIR=${IGN_GAZEBO_PLUGIN_INSTALL_DIR}:/usr/local/lib/scenario/plugins

Testing

Run the pendulum Python example from VSCode. Set up a launch configuration entry in ~/robotology/.vscode/launch.json: 

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: ScenarIO",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "env": {
        "IGN_VERBOSE": "0",
        "IGN_CONFIG_PATH": "/usr/local/share/ignition",
        "IGN_RENDERING_RESOURCE_PATH": "/usr/local/share/ignition/ignition-rendering7",
      },
    },
  ]
}

Add the following lines to the pendulum example to ensure that ignition uses the brew installed version of ruby:

# ensure we are using /usr/local/opt/ruby/bin/ruby instead of the system version
import os
PATH = os.getenv("PATH")
PATH = "/usr/local/opt/ruby/bin:" + PATH 
os.environ["PATH"] = PATH

Then run the example:

scenario_pendulum_vscode

scenario_pendulum_gazebo

diegoferigo commented 2 years ago

Wow @srmainwaring great work, nicely done! Thanks for this big effort to smooth the macOS experience. I don't have a macOS machine to test everything myself, but I will eagerly wait for your contributions to land upstream and then giving another shot in the conda-forge packaging I drafted in https://github.com/conda-forge/staged-recipes/pull/16582 (unfortunately I'm a bit busy lately and I have to postpone it a little longer). As you noted, from the ScenarIO point of view, changes are minimal, only the introduction of Garden support was missing. I can surely add it to the nightly channel so that your setup could get rid of a fork, and those that are interested in building the project from source can benefit from it! And, actually, note to myself, it would a good idea in the future adding a similar CMake support of the development Ignition distribution right after the previous upstream stable version is released.