Clean UP & Dockerization of eos74bo

[GemmaTuron]

Hi @emmakodes Please check that the repository is up to the latest version of the workflows, structure etc

[emmakodes]

Hello, @GemmaTuron this model is ready, you can review it.

[GemmaTuron]

Hi @emmakodes !

Seems the changes do not pass the Docker Building, probably the build in ARM64 is failing, please have a look at the Action and try to see what failed

[emmakodes]

Sure @GemmaTuron, on it.

[GemmaTuron]

Hi @emmakodes

Do you have more info on why the Docker Build is failing in ARM64?

[emmakodes]

Hello @GemmaTuron Docker Build fails on the Build and push job. Specifically when it executes [linux/arm64 2/2] RUN ersilia -v fetch eos74bo

I can see that it failed to build wheels for scikit-learn:

#8 598.5 Building wheels for collected packages: scikit-learn
#8 598.5   Building wheel for scikit-learn (setup.py): started
#8 598.5   Building wheel for scikit-learn (setup.py): finished with status 'error'
#8 598.5   error: subprocess-exited-with-error
#8 598.5   
#8 598.5   × python setup.py bdist_wheel did not run successfully.
#8 598.5   │ exit code: 1
#8 598.5   ╰─> [23 lines of output]
#8 598.5       Partial import of sklearn during the build process.
#8 598.5       Traceback (most recent call last):
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 195, in check_package_status
#8 598.5           module = importlib.import_module(package)
#8 598.5         File "/usr/bin/conda/envs/eos74bo/lib/python3.7/importlib/__init__.py", line 127, in import_module
#8 598.5           return _bootstrap._gcd_import(name[level:], package, level)
#8 598.5         File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
#8 598.5         File "<frozen importlib._bootstrap>", line 983, in _find_and_load
#8 598.5         File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
#8 598.5       ModuleNotFoundError: No module named 'scipy'
#8 598.5       Traceback (most recent call last):
#8 598.5         File "<string>", line 36, in <module>
#8 598.5         File "<pip-setuptools-caller>", line 34, in <module>
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 303, in <module>
#8 598.5           setup_package()
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 293, in setup_package
#8 598.5           check_package_status('scipy', SCIPY_MIN_VERSION)
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 221, in check_package_status
#8 598.5           .format(package, req_str, instructions))
#8 598.5       ImportError: scipy is not installed.
#8 598.5       scikit-learn requires scipy >= 0.17.0.
#8 598.5       Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html
#8 598.5       
#8 598.5       [end of output]
#8 598.5   
#8 598.5   note: This error originates from a subprocess, and is likely not a problem with pip.
#8 598.5   ERROR: Failed building wheel for scikit-learn
#8 598.5   Running setup.py clean for scikit-learn
#8 598.5 Failed to build scikit-learn
#8 598.5 Installing collected packages: scipy, joblib, scikit-learn
#8 598.5   Running setup.py install for scikit-learn: started
#8 598.5   Running setup.py install for scikit-learn: finished with status 'error'
#8 598.5   error: subprocess-exited-with-error
#8 598.5   
#8 598.5   × Running setup.py install for scikit-learn did not run successfully.
#8 598.5   │ exit code: 1
#8 598.5   ╰─> [22 lines of output]
#8 598.5       Partial import of sklearn during the build process.
#8 598.5       Traceback (most recent call last):
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/sklearn/_build_utils/__init__.py", line 32, in _check_cython_version
#8 598.5           import Cython
#8 598.5       ModuleNotFoundError: No module named 'Cython'
#8 598.5       
#8 598.5       During handling of the above exception, another exception occurred:
#8 598.5       
#8 598.5       Traceback (most recent call last):
#8 598.5         File "<string>", line 36, in <module>
#8 598.5         File "<pip-setuptools-caller>", line 34, in <module>
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 303, in <module>
#8 598.5           setup_package()
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 299, in setup_package
#8 598.5           setup(**metadata)
#8 598.5         File "/usr/bin/conda/envs/eos74bo/lib/python3.7/site-packages/numpy/distutils/core.py", line 135, in setup
#8 598.5           config = configuration()
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 180, in configuration
#8 598.5           _check_cython_version()
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/sklearn/_build_utils/__init__.py", line 35, in _check_cython_version
#8 598.5           raise ModuleNotFoundError(message)
#8 598.5       ModuleNotFoundError: Please install Cython with a version >= 0.28.5 in order to build a scikit-learn from source.
#8 598.5       [end of output]
#8 598.5   
#8 598.5   note: This error originates from a subprocess, and is likely not a problem with pip.
#8 598.5 error: legacy-install-failure

[GemmaTuron]

Hello @emmakodes

Please, look in detail on the Action so you can identify the REAL source of the error, the fetch command is just general. In this particular case, I see the following failing. Instead of bypassing the ARM64 install, we should find a sckitlearn install that works for both AMD64 and ARM64 if possible. Have a look at the error and try if the model works with the required scikitleanr and scipy version

#8 598.5 Building wheels for collected packages: scikit-learn
#8 598.5   Building wheel for scikit-learn (setup.py): started
#8 598.5   Building wheel for scikit-learn (setup.py): finished with status 'error'
#8 598.5   error: subprocess-exited-with-error
#8 598.5   
#8 598.5   × python setup.py bdist_wheel did not run successfully.
#8 598.5   │ exit code: 1
#8 598.5   ╰─> [23 lines of output]
#8 598.5       Partial import of sklearn during the build process.
#8 598.5       Traceback (most recent call last):
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 195, in check_package_status
#8 598.5           module = importlib.import_module(package)
#8 598.5         File "/usr/bin/conda/envs/eos74bo/lib/python3.7/importlib/__init__.py", line 127, in import_module
#8 598.5           return _bootstrap._gcd_import(name[level:], package, level)
#8 598.5         File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
#8 598.5         File "<frozen importlib._bootstrap>", line 983, in _find_and_load
#8 598.5         File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
#8 598.5       ModuleNotFoundError: No module named 'scipy'
#8 598.5       Traceback (most recent call last):
#8 598.5         File "<string>", line 36, in <module>
#8 598.5         File "<pip-setuptools-caller>", line 34, in <module>
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 303, in <module>
#8 598.5           setup_package()
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 293, in setup_package
#8 598.5           check_package_status('scipy', SCIPY_MIN_VERSION)
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 221, in check_package_status
#8 598.5           .format(package, req_str, instructions))
#8 598.5       ImportError: scipy is not installed.
#8 598.5       scikit-learn requires scipy >= 0.17.0.
#8 598.5       Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html
#8 598.5       
#8 598.5       [end of output]
#8 598.5   
#8 598.5   note: This error originates from a subprocess, and is likely not a problem with pip.
#8 598.5   ERROR: Failed building wheel for scikit-learn
#8 598.5   Running setup.py clean for scikit-learn
#8 598.5 Failed to build scikit-learn
#8 598.5 Installing collected packages: scipy, joblib, scikit-learn
#8 598.5   Running setup.py install for scikit-learn: started
#8 598.5   Running setup.py install for scikit-learn: finished with status 'error'
#8 598.5   error: subprocess-exited-with-error
#8 598.5   
#8 598.5   × Running setup.py install for scikit-learn did not run successfully.
#8 598.5   │ exit code: 1
#8 598.5   ╰─> [22 lines of output]
#8 598.5       Partial import of sklearn during the build process.
#8 598.5       Traceback (most recent call last):
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/sklearn/_build_utils/__init__.py", line 32, in _check_cython_version
#8 598.5           import Cython
#8 598.5       ModuleNotFoundError: No module named 'Cython'
#8 598.5       
#8 598.5       During handling of the above exception, another exception occurred:
#8 598.5       
#8 598.5       Traceback (most recent call last):
#8 598.5         File "<string>", line 36, in <module>
#8 598.5         File "<pip-setuptools-caller>", line 34, in <module>
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 303, in <module>
#8 598.5           setup_package()
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 299, in setup_package
#8 598.5           setup(**metadata)
#8 598.5         File "/usr/bin/conda/envs/eos74bo/lib/python3.7/site-packages/numpy/distutils/core.py", line 135, in setup
#8 598.5           config = configuration()
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/setup.py", line 180, in configuration
#8 598.5           _check_cython_version()
#8 598.5         File "/tmp/pip-install-w8v2c_jf/scikit-learn_929c138c51424063822c436783eb79ba/sklearn/_build_utils/__init__.py", line 35, in _check_cython_version
#8 598.5           raise ModuleNotFoundError(message)
#8 598.5       ModuleNotFoundError: Please install Cython with a version >= 0.28.5 in order to build a scikit-learn from source.
#8 598.5       [end of output]
#8 598.5   
#8 598.5   note: This error originates from a subprocess, and is likely not a problem with pip.
#8 598.5 error: legacy-install-failure
#8 598.5 
#8 598.5 × Encountered error while trying to install package.
#8 598.5 ╰─> scikit-learn

[GemmaTuron]

thanks for looking into it. See my comment above just published at the same time

[emmakodes]

Okay thanks @GemmaTuron let me do that now

[emmakodes]

Hello @GemmaTuron I discovered here: Wheel support for aarch64 | here and here that for users to get a working installation of scikit-learn on arm64 they have to install scikit-learn and its dependencies from the conda-forge distribution using the miniforge or mambaforge installers. I have implemented scikit-learn to install from conda-forge distribution in the new pull request.

[GemmaTuron]

Hi @emmakodes !

The Docker build is failing, seems torch cannot be installed

[emmakodes]

Yes @GemmaTuron #8 1541.6 ERROR: Could not find a version that satisfies the requirement torch==1.8.0+cpu (from versions: 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1) I changed to torch 1.9.0 in the new pull request as suggested by @leoank

[GemmaTuron]

Hi @emmakodes The latest Docker Build still failed, did you have time to look at the Action to see what was failing? Also, on a quick look, it seemed to be installing torch 1.8 still? python -m pip --disable-pip-version-check install torch==1.8.0+cpu torchvision==0.9.0+cpu torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html Please have a look

[emmakodes]

I now see Docker build still fails with torch==1.9.0+cpu ```ERROR: Could not find a version that satisfies the requirement torch==1.9.0+cpu (from versions: 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1)

8 1630.7 ERROR: No matching distribution found for torch==1.9.0+cpu```

[GemmaTuron]

Hi @emmakodes

Please can you look for more information on the error? why is there no version for 1.9+cpu? Is AMD64 working with this or are both failing?

[miquelduranfrigola]

Hi @emmakodes - what is the current status of this?

[emmakodes]

Hello @miquelduranfrigola docker build fails for torch==1.9.0+cpu on linux/arm64 since there is no version of such on linux/arm64

[miquelduranfrigola]

Thanks @emmakodes - this is useful.

@GemmaTuron - I would suggest that we update the workflows and see what happens with this model. What is your take?

[GemmaTuron]

I have updated the workflows @emmakodes please keep an eye on them and let us know if all the tests are working!

[emmakodes]

Hello, @GemmaTuron the workflow is successful. upload-model-to-dockerhub only build for linux/amd64. I still need to look into this new workflow to understand how they work.