Unable to install medaka on Mac M3

[sandeepkasaragod]

Hi Team,

I am unable to install the Medaka on Mac M3. Here are my system details and the error I get when I am compiling.

machdep.cpu.cores_per_package: 11 machdep.cpu.core_count: 11 machdep.cpu.logical_per_package: 11 machdep.cpu.thread_count: 11 machdep.cpu.brand_string: Apple M3 Pro

ProductName: macOS ProductVersion: 14.3.1 BuildVersion: 23D60

$make install Installing prerequisites with homebrew brew install pkg-config hdf5@1.10 openssl@3 ==> Downloading https://formulae.brew.sh/api/formula.jws.json ###################################################################################################################################################################### 100.0% ==> Downloading https://formulae.brew.sh/api/cask.jws.json ###################################################################################################################################################################### 100.0% Warning: pkg-config 0.29.2_3 is already installed and up-to-date. To reinstall 0.29.2_3, run: brew reinstall pkg-config Warning: hdf5@1.10 1.10.11 is already installed and up-to-date. To reinstall 1.10.11, run: brew reinstall hdf5@1.10 Warning: openssl@3 3.2.1 is already installed and up-to-date. To reinstall 3.2.1, run: brew reinstall openssl@3 Compiling and installing mappy from custom repository . ./venv/bin/activate && pip install git+https://github.com/cjw85/minimap2.git@1951fe908ae83efd4f459e060d29b21254d54993 Collecting git+https://github.com/cjw85/minimap2.git@1951fe908ae83efd4f459e060d29b21254d54993 Cloning https://github.com/cjw85/minimap2.git (to revision 1951fe908ae83efd4f459e060d29b21254d54993) to /private/var/folders/7z/5gjchrpn52s3gxfz062y6q940000gq/T/pip-req-build-5w1blv0e Running command git clone --filter=blob:none --quiet https://github.com/cjw85/minimap2.git /private/var/folders/7z/5gjchrpn52s3gxfz062y6q940000gq/T/pip-req-build-5w1blv0e Running command git rev-parse -q --verify 'sha^1951fe908ae83efd4f459e060d29b21254d54993' Running command git fetch -q https://github.com/cjw85/minimap2.git 1951fe908ae83efd4f459e060d29b21254d54993 Running command git checkout -q 1951fe908ae83efd4f459e060d29b21254d54993 Resolved https://github.com/cjw85/minimap2.git to commit 1951fe908ae83efd4f459e060d29b21254d54993 Running command git submodule update --init --recursive -q error: RPC failed; curl 18 Transferred a partial file error: 2515 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output fatal: clone of 'https://github.com/nemequ/simde.git' into submodule path '/private/var/folders/7z/5gjchrpn52s3gxfz062y6q940000gq/T/pip-req-build-5w1blv0e/lib/simde' failed Failed to clone 'lib/simde'. Retry scheduled error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) error: 4006 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output fatal: clone of 'https://github.com/nemequ/simde.git' into submodule path '/private/var/folders/7z/5gjchrpn52s3gxfz062y6q940000gq/T/pip-req-build-5w1blv0e/lib/simde' failed Failed to clone 'lib/simde' a second time, aborting error: subprocess-exited-with-error

× git submodule update --init --recursive -q did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× git submodule update --init --recursive -q did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. make: *** [pyprep-m1] Error 1

[cjw85]

The error suggests your computer is having difficulty downloading content from GitHub, e.g.:

Running command git submodule update --init --recursive -q
error: RPC failed; curl 18 Transferred a partial file
error: 2515 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

I don't know that there is anything I can suggest to help here.

[sandeepkasaragod]

Thank you for your response. I have downgraded medaka to 1.8.2. I checked the installation by typing medaka --version, the command worked fine. However, when I run medaka consensus with input file, it displayed error saying "illegal instruction: 4".

[cjw85]

I imagine what has happened here is you have managed to install a x86 package of medaka (hence the "illegal instruction" error).

Typically this happens to people on macOS when they have followed ill-advised instructions that are around the internet to fool their shell, conda, or Python environment into thinking they are running on an x86 processor rather than ARM.

[sandeepkasaragod]

Hi Chris, Thank you for your response. I have used standard way of installing medaka using pip. Here is my command.

pip install medaka==1.8.2

Do you have any suggestion installing medaka on Mac.

[cjw85]

Installing with pip install medaka==1.8.2 on a macOS device should result in the source distribution being pulled and the package being compiled much as your initial attempt with make install.

Without seeing full logs from the pip install I can't suggest a reason why you have ended up managing to get a binary that appears to be compiled for x86 rather than ARM.

[sandeepkasaragod]

Hi Chris,

Here is the log details.

Python version 3.8.18

$pip install medaka==1.8.2 Processing /Users/sandeep.kasaragod/Library/Caches/pip/wheels/07/e0/d2/03a7007704df81b31c955d4b7db92333d54431215ae4bb6de9/medaka-1.8.2-cp38-cp38-macosx_10_9_x86_64.whl Collecting ont-fast5-api Using cached ont_fast5_api-4.1.3-py3-none-any.whl (2.3 MB) Processing /Users/sandeep.kasaragod/Library/Caches/pip/wheels/cf/7e/ba/d04a8d00c3007add72182970891a977e576bb46a5a2a276ea0/grpcio-1.62.0-cp38-cp38-macosx_10_10_x86_64.whl Processing /Users/sandeep.kasaragod/Library/Caches/pip/wheels/8b/83/b1/9898eb223680ff77293ce785cfa2be9410d36853b6d5cd5973/pyspoa-0.2.1-cp38-cp38-macosx_10_9_x86_64.whl Collecting h5py Using cached h5py-3.10.0-cp38-cp38-macosx_10_9_x86_64.whl (3.2 MB) Processing /Users/sandeep.kasaragod/Library/Caches/pip/wheels/45/23/de/5789a92962483fd33cb06674792b9697c1b3766d7c7742830e/intervaltree-3.1.0-py2.py3-none-any.whl Collecting tensorflow~=2.10.0 Using cached tensorflow-2.10.1-cp38-cp38-macosx_10_14_x86_64.whl (241.2 MB) Processing /Users/sandeep.kasaragod/Library/Caches/pip/wheels/b0/8d/d5/b442347c2e20052ecc68f8c1f75c66ab18c03dd4f5a5f7f2ce/mappy-2.26-cp38-cp38-macosx_10_9_x86_64.whl Collecting pysam>=0.16.0.1 Using cached pysam-0.22.0-cp38-cp38-macosx_10_9_x86_64.whl (8.4 MB) Collecting numpy>=1.21.6 Using cached numpy-1.24.4-cp38-cp38-macosx_10_9_x86_64.whl (19.8 MB) Collecting parasail Using cached parasail-1.3.4-py2.py3-none-macosx_10_9_x86_64.whl (3.2 MB) Collecting requests Using cached requests-2.31.0-py3-none-any.whl (62 kB) Collecting cffi==1.15.0 Using cached cffi-1.15.0-cp38-cp38-macosx_10_9_x86_64.whl (178 kB) Collecting edlib Using cached edlib-1.3.9-cp38-cp38-macosx_10_9_x86_64.whl (64 kB) Requirement already satisfied: setuptools in /Users/sandeep.kasaragod/miniforge3/envs/medaka/lib/python3.8/site-packages (from ont-fast5-api->medaka==1.8.2) (65.3.0) Collecting packaging Using cached packaging-23.2-py3-none-any.whl (53 kB) Processing /Users/sandeep.kasaragod/Library/Caches/pip/wheels/29/a7/e5/80302edade245519d68bbdf44f7879049f9b67e7bae7ed66e4/progressbar33-2.4-py3-none-any.whl Collecting sortedcontainers<3.0,>=2.0 Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB) Collecting libclang>=13.0.0 Using cached libclang-16.0.6-py2.py3-none-macosx_10_9_x86_64.whl (24.5 MB) Collecting gast<=0.4.0,>=0.2.1 Using cached gast-0.4.0-py3-none-any.whl (9.8 kB) Collecting keras<2.11,>=2.10.0 Using cached keras-2.10.0-py2.py3-none-any.whl (1.7 MB) Collecting opt-einsum>=2.3.2 Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB) Collecting flatbuffers>=2.0 Downloading flatbuffers-24.3.7-py2.py3-none-any.whl (26 kB) Collecting termcolor>=1.1.0 Using cached termcolor-2.4.0-py3-none-any.whl (7.7 kB) Collecting keras-preprocessing>=1.1.1 Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB) Collecting google-pasta>=0.1.1 Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB) Collecting tensorboard<2.11,>=2.10 Using cached tensorboard-2.10.1-py3-none-any.whl (5.9 MB) Collecting protobuf<3.20,>=3.9.2 Using cached protobuf-3.19.6-cp38-cp38-macosx_10_9_x86_64.whl (980 kB) Collecting tensorflow-estimator<2.11,>=2.10.0 Using cached tensorflow_estimator-2.10.0-py2.py3-none-any.whl (438 kB) Collecting typing-extensions>=3.6.6 Using cached typing_extensions-4.10.0-py3-none-any.whl (33 kB) Collecting tensorflow-io-gcs-filesystem>=0.23.1 Using cached tensorflow_io_gcs_filesystem-0.34.0-cp38-cp38-macosx_10_14_x86_64.whl (1.7 MB) Collecting six>=1.12.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting wrapt>=1.11.0 Using cached wrapt-1.16.0-cp38-cp38-macosx_10_9_x86_64.whl (37 kB) Collecting absl-py>=1.0.0 Using cached absl_py-2.1.0-py3-none-any.whl (133 kB) Collecting astunparse>=1.6.0 Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB) Collecting charset-normalizer<4,>=2 Using cached charset_normalizer-3.3.2-cp38-cp38-macosx_10_9_x86_64.whl (121 kB) Collecting urllib3<3,>=1.21.1 Using cached urllib3-2.2.1-py3-none-any.whl (121 kB) Collecting certifi>=2017.4.17 Using cached certifi-2024.2.2-py3-none-any.whl (163 kB) Collecting idna<4,>=2.5 Using cached idna-3.6-py3-none-any.whl (61 kB) Collecting pycparser Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB) Collecting google-auth<3,>=1.6.3 Using cached google_auth-2.28.1-py2.py3-none-any.whl (186 kB) Collecting tensorboard-data-server<0.7.0,>=0.6.0 Using cached tensorboard_data_server-0.6.1-py3-none-macosx_10_9_x86_64.whl (3.5 MB) Collecting markdown>=2.6.8 Using cached Markdown-3.5.2-py3-none-any.whl (103 kB) Requirement already satisfied: wheel>=0.26 in /Users/sandeep.kasaragod/miniforge3/envs/medaka/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow~=2.10.0->medaka==1.8.2) (0.34.2) Collecting werkzeug>=1.0.1 Using cached werkzeug-3.0.1-py3-none-any.whl (226 kB) Collecting google-auth-oauthlib<0.5,>=0.4.1 Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB) Collecting tensorboard-plugin-wit>=1.6.0 Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB) Collecting cachetools<6.0,>=2.0.0 Using cached cachetools-5.3.3-py3-none-any.whl (9.3 kB) Collecting rsa<5,>=3.1.4 Using cached rsa-4.9-py3-none-any.whl (34 kB) Collecting pyasn1-modules>=0.2.1 Using cached pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB) Collecting importlib-metadata>=4.4; python_version < "3.10" Using cached importlib_metadata-7.0.2-py3-none-any.whl (24 kB) Collecting MarkupSafe>=2.1.1 Using cached MarkupSafe-2.1.5-cp38-cp38-macosx_10_9_x86_64.whl (14 kB) Collecting requests-oauthlib>=0.7.0 Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB) Collecting pyasn1>=0.1.3 Using cached pyasn1-0.5.1-py2.py3-none-any.whl (84 kB) Collecting zipp>=0.5 Using cached zipp-3.17.0-py3-none-any.whl (7.4 kB) Collecting oauthlib>=3.0.0 Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB) Installing collected packages: packaging, progressbar33, numpy, h5py, ont-fast5-api, grpcio, pyspoa, sortedcontainers, intervaltree, libclang, gast, keras, opt-einsum, flatbuffers, termcolor, six, keras-preprocessing, google-pasta, cachetools, pyasn1, rsa, pyasn1-modules, google-auth, tensorboard-data-server, charset-normalizer, urllib3, certifi, idna, requests, zipp, importlib-metadata, markdown, MarkupSafe, werkzeug, absl-py, oauthlib, requests-oauthlib, google-auth-oauthlib, tensorboard-plugin-wit, protobuf, tensorboard, tensorflow-estimator, typing-extensions, tensorflow-io-gcs-filesystem, wrapt, astunparse, tensorflow, mappy, pysam, parasail, pycparser, cffi, edlib, medaka Successfully installed MarkupSafe-2.1.5 absl-py-2.1.0 astunparse-1.6.3 cachetools-5.3.3 certifi-2024.2.2 cffi-1.15.0 charset-normalizer-3.3.2 edlib-1.3.9 flatbuffers-24.3.7 gast-0.4.0 google-auth-2.28.1 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.62.0 h5py-3.10.0 idna-3.6 importlib-metadata-7.0.2 intervaltree-3.1.0 keras-2.10.0 keras-preprocessing-1.1.2 libclang-16.0.6 mappy-2.26 markdown-3.5.2 medaka-1.8.2 numpy-1.24.4 oauthlib-3.2.2 ont-fast5-api-4.1.3 opt-einsum-3.3.0 packaging-23.2 parasail-1.3.4 progressbar33-2.4 protobuf-3.19.6 pyasn1-0.5.1 pyasn1-modules-0.3.0 pycparser-2.21 pysam-0.22.0 pyspoa-0.2.1 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 sortedcontainers-2.4.0 tensorboard-2.10.1 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.10.1 tensorflow-estimator-2.10.0 tensorflow-io-gcs-filesystem-0.34.0 termcolor-2.4.0 typing-extensions-4.10.0 urllib3-2.2.1 werkzeug-3.0.1 wrapt-1.16.0 zipp-3.17.0

[sandeepkasaragod]

After your suggestions, I just ran this

python
import platform
>>> platform.machine()
'x86_64'

Here is the catch, the conda has downloaded x86_64 instead arm64. Let me download the correct version of python and see how it works. Thanks for your great help.

[cjw85]

The way I would recommend to you to set up medaka on a macOS device is through the make install route you were trying initially. I've just noticed that pip install medaka won't work on a macOS device because the dependencymappy will not correctly compile. (make install works around this by using a fork of mappy).

Before running be sure than the python3 found in your environment is not an x64 version (from conda or anywhere else), but rather the standard (ARM) one from the system.

[sandeepkasaragod]

Thanks for the suggestions, this methods worked on my system.