Failed to build onnx on MacOS when installing ml-agents

[neilmendoza]

Describe the bug Lots of errors stemming from current dependency versions in setup.py making pip try to compile onnx from source...

Error 1: Could not find cmake executable! Attempted Resolution 1: brew install cmake

Error 2: Protobuf compiler not found Attempted Resolution 2: brew install protobuf

Error 3: _In file included from /opt/homebrew/include/absl/base/config.h:86: /opt/homebrew/include/absl/base/policychecks.h:79:2: error: "C++ versions less than C++14 are not supported." Resolution 3: Give up on dependency versions specified as they are forcing onnx to be built from source, uninstall cmake and protobuf installed above, modify setup.py files instead and cross fingers everything still works. I ran one training and didn't crash.

diff --git a/ml-agents-envs/setup.py b/ml-agents-envs/setup.py
index 3fa691543..3cb3e33d3 100644
--- a/ml-agents-envs/setup.py
+++ b/ml-agents-envs/setup.py
@@ -56,7 +56,7 @@ setup(
         "cloudpickle",
         "grpcio>=1.11.0,<=1.48.2",
         "Pillow>=4.2.1",
-        "protobuf>=3.6,<3.20",
+        "protobuf>=3.6,<=3.20.2",
         "pyyaml>=3.1.0",
         "gym>=0.21.0",
         "pettingzoo==1.15.0",
diff --git a/ml-agents/setup.py b/ml-agents/setup.py
index f77374d07..8f528ae14 100644
--- a/ml-agents/setup.py
+++ b/ml-agents/setup.py
@@ -60,7 +60,7 @@ setup(
         f"mlagents_envs=={VERSION}",
         "numpy>=1.13.3,<2.0",
         "Pillow>=4.2.1",
-        "protobuf>=3.6,<3.20",
+        "protobuf>=3.6,<=3.20.2",
         "pyyaml>=3.1.0",
         "torch>=1.13.1",
         "tensorboard>=2.14",
@@ -72,7 +72,7 @@ setup(
         "attrs>=19.3.0",
         "huggingface_hub>=0.14",
         'pypiwin32==223;platform_system=="Windows"',
-        "onnx==1.12.0",
+        "onnx==1.13.0",
     ],
     python_requires=">=3.10.1,<=3.10.12",
     entry_points={

To Reproduce Steps to reproduce the behavior: pip install ./ml-agents

Environment (please complete the following information):

• Unity Version: Unity 2022.3.13f1
• OS + version: MacOS 14.0
• ML-Agents version: release_21
• Torch version: 2.1.1
• Environment: virtual env with Python 3.10.12

[Ale1]

Mac OS 13.5 user here. Had same issue with ONNX yesterday and your suggested fixes worked for me.

P.S - I also had uninstall the grpcio installed through pip since it brings an incomptible version with M2 macs, and instead install a different variant with anaconda conda install grpcio (with channel set to conda-forge) instead. But that seems to be a separate issue.

[kkkkkom333]

Mac OS 13.5 user here. Had same issue with ONNX yesterday and your suggested fixes worked for me.

P.S - I also had uninstall the grpcio installed through pip since it brings an incomptible version with M2 macs, and instead install a different variant with anaconda conda install grpcio (with channel set to conda-forge) instead. But that seems to be a separate issue.

conda install grpcio helped!

[neilmendoza]

If you installed grpcio to get protoc in order to compile ONNX, I don't think it's necessary anymore with the above changes to setup.py as seems like pip then uses a pre-compiled ONNX. I'm not using conda though, just pip in a pyenv-virtualenv, so maybe your setup is different.

[miguelalonsojr]

Please try the MLA setup with the develop branch of the repo instead of the release 21 branch. It's been fixed on develop. Please let me know if it worked out.

[ChristianPickettCode]

Faced similar issue, also had onnx related issues when trying to stop training and save (ctrl-c). What work for me was: pip install onnx==1.13. Saw errors : "ERROR: pip's dependency resolver does not currently..." but onnx successfully installed and now the models save (onnx files get created).

[josmuniz]

I think that I have the same issues but Mac 14.2. I try to install it directly : (mlagents) josemuniz@MacBook-Pro-2 ml-agents % pip install "onnx==1.13.0" pip install "mlagents-envs==1.0.0" pip install "protobuf==3.20.2"

but there are incompatibility between recommended versions.

Note: Collecting onnx==1.13.0 Using cached onnx-1.13.0-cp310-cp310-macosx_10_12_universal2.whl (14.0 MB) Requirement already satisfied: numpy>=1.16.6 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from onnx==1.13.0) (1.21.2) Requirement already satisfied: protobuf<4,>=3.20.2 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from onnx==1.13.0) (3.20.2) Requirement already satisfied: typing-extensions>=3.6.2.1 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from onnx==1.13.0) (4.9.0) Installing collected packages: onnx Attempting uninstall: onnx Found existing installation: onnx 1.15.0 Uninstalling onnx-1.15.0: Successfully uninstalled onnx-1.15.0 Successfully installed onnx-1.13.0 Requirement already satisfied: mlagents-envs==1.0.0 in ./ml-agents-envs (1.0.0) Requirement already satisfied: cloudpickle in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from mlagents-envs==1.0.0) (3.0.0) Requirement already satisfied: grpcio<=1.48.2,>=1.11.0 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from mlagents-envs==1.0.0) (1.48.2) Requirement already satisfied: Pillow>=4.2.1 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from mlagents-envs==1.0.0) (10.1.0) Collecting protobuf<3.20,>=3.6 (from mlagents-envs==1.0.0) Using cached protobuf-3.19.6-py2.py3-none-any.whl (162 kB) Requirement already satisfied: pyyaml>=3.1.0 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from mlagents-envs==1.0.0) (6.0.1) Requirement already satisfied: gym>=0.21.0 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from mlagents-envs==1.0.0) (0.26.2) Requirement already satisfied: pettingzoo==1.15.0 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from mlagents-envs==1.0.0) (1.15.0) Requirement already satisfied: numpy==1.21.2 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from mlagents-envs==1.0.0) (1.21.2) Requirement already satisfied: filelock>=3.4.0 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from mlagents-envs==1.0.0) (3.13.1) Requirement already satisfied: six>=1.5.2 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from grpcio<=1.48.2,>=1.11.0->mlagents-envs==1.0.0) (1.16.0) Requirement already satisfied: gym-notices>=0.0.4 in /Users/josemuniz/miniforge3/envs/mlagents/lib/python3.10/site-packages (from gym>=0.21.0->mlagents-envs==1.0.0) (0.0.8) Installing collected packages: protobuf Attempting uninstall: protobuf Found existing installation: protobuf 3.20.2 Uninstalling protobuf-3.20.2: Successfully uninstalled protobuf-3.20.2 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. onnx 1.13.0 requires protobuf<4,>=3.20.2, but you have protobuf 3.19.6 which is incompatible. Successfully installed protobuf-3.19.6 Collecting protobuf==3.20.2 Using cached protobuf-3.20.2-py2.py3-none-any.whl (162 kB) Installing collected packages: protobuf Attempting uninstall: protobuf Found existing installation: protobuf 3.19.6 Uninstalling protobuf-3.19.6: Successfully uninstalled protobuf-3.19.6 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. mlagents-envs 1.0.0 requires protobuf<3.20,>=3.6, but you have protobuf 3.20.2 which is incompatible. Successfully installed protobuf-3.20.2

[neilmendoza]

@josmuniz The three changes to ml-agents-envs/setup.py that I put in the original post in this thread should solve that issue.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale. Please open a new issue for related bugs.

[yvonmeunier]

bless this thread it actually worked

[cihadturhan]

Wow finally a solution! Thanks @neilmendoza. This should be updated in v21!