bidscoin pip install fails on apple silicon

[dkp]

Describe the bug bidscoin environment creation fails on Apple Silicon (though it worked on my intel mac).

The install hangs here:

Collecting PyQt5>=5.12.1 (from bidscoin)
  Using cached PyQt5-5.15.9.tar.gz (3.2 MB)
  Installing build dependencies ... done

StackOverflow reports that pyqt5 is not compatible with the ARM architecture, but PyQt6 IS compatible: https://stackoverflow.com/questions/71046800/how-to-install-pyqt5-on-m1-arm64-architecture

To reproduce On Apple Silicon:

conda create -n bidscoin python=3.10
conda activate bidscoin
pip install bidscoin[all] 

Expected behavior I hoped it would install ; (

[marcelzwiers]

Mhhh, I did make a port to qt6 some time ago, but I didn't make it default because it looked less good (on Linux). Can you test to see if that would work for you?

---

Sent from my phone

Op vr 30 jun. 2023 03:31 schreef dkp @.***>:

Describe the bug bidscoin environment creation fails on Apple Silicon (though it worked on my intel mac).

The install hangs here:

Collecting PyQt5>=5.12.1 (from bidscoin) Using cached PyQt5-5.15.9.tar.gz (3.2 MB) Installing build dependencies ... done

StackOverflow reports that pyqt5 is not compatible with the ARM architecture, but pyqt6 IS compatible.

https://stackoverflow.com/questions/71046800/how-to-install-pyqt5-on-m1-arm64-architecture

To reproduce On Apple Silicon:

conda create -n bidscoin python=3.10 conda activate bidscoin pip install bidscoin[all]

Expected behavior I hoped it would install ; (

— Reply to this email directly, view it on GitHub https://github.com/Donders-Institute/bidscoin/issues/189, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTUGL6RTAQL3OWGBXWKWCLXNYT5VANCNFSM6AAAAAAZZG2NNI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[marcelzwiers]

See the qt6 branch comments in this issue:

https://github.com/Donders-Institute/bidscoin/issues/131

---

Sent from my phone

Op vr 30 jun. 2023 03:31 schreef dkp @.***>:

Describe the bug bidscoin environment creation fails on Apple Silicon (though it worked on my intel mac).

The install hangs here:

Collecting PyQt5>=5.12.1 (from bidscoin) Using cached PyQt5-5.15.9.tar.gz (3.2 MB) Installing build dependencies ... done

StackOverflow reports that pyqt5 is not compatible with the ARM architecture, but pyqt6 IS compatible.

https://stackoverflow.com/questions/71046800/how-to-install-pyqt5-on-m1-arm64-architecture

To reproduce On Apple Silicon:

conda create -n bidscoin python=3.10 conda activate bidscoin pip install bidscoin[all]

Expected behavior I hoped it would install ; (

— Reply to this email directly, view it on GitHub https://github.com/Donders-Institute/bidscoin/issues/189, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTUGL6RTAQL3OWGBXWKWCLXNYT5VANCNFSM6AAAAAAZZG2NNI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[dkp]

Wow. I think it may just have worked!

python setup.py install
...
Finished processing dependencies for bidscoin==4.0.0

pyqt6_issues.txt

bidscoin gives me a nice usage message, however, when I try to start the GUI, I run into a problem with PyQT ; ( I have attached the messages output by bidscoin -t However, I also tried to run the gui and it crashed with complaints about the same library.

[marcelzwiers]

Could it be that you have both PyQt5 and PyQt6 installed? Or did you install the qt6 branch in it's own new environment?

[marcelzwiers]

I think this library error is a bit out of my scope, @bendhouseart have you perhaps seen this?

[dkp]

My Python knowledge is wobbly. Maybe you can help me understand where to go from here. Honestly, i can't say in PyQt5 is lurking somewhere.

I cloned the qt6 branch:

git clone --branch qt6 https://github.com/Donders-Institute/bidscoin.git

I tried creating a venv and building there with python3 setup.py install Although it claims to finish, it is clearly has problems:

(venv_qt6) [lolo:bidscoin_qt6:qt6 586] $ bidscoin -t
Traceback (most recent call last):
  File "/Users/dpat/projects/Testing/neuroimaging/dicom_conversion/bidscoin/code/venv_qt6/bin/bidscoin", line 33, in <module>
    sys.exit(load_entry_point('bidscoin==4.0.0', 'console_scripts', 'bidscoin')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dpat/projects/Testing/neuroimaging/dicom_conversion/bidscoin/code/venv_qt6/bin/bidscoin", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/dpat/projects/Testing/neuroimaging/dicom_conversion/bidscoin/code/venv_qt6/lib/python3.11/site-packages/bidscoin-4.0.0-py3.11.egg/bidscoin/bcoin.py", line 33, in <module>
    from ruamel.yaml import YAML
ImportError: cannot import name 'YAML' from 'ruamel.yaml' (/Users/dpat/projects/Testing/neuroimaging/dicom_conversion/bidscoin/code/venv_qt6/lib/python3.11/site-packages/ruamel.yaml.clib-0.2.7-py3.11-macosx-13-arm64.egg/ruamel/yaml/__init__.py)

Outside a virtual env has the same problems. This is installing with brew's python, so I uninstalled:

Found existing installation: bidscoin 4.0.0
Uninstalling bidscoin-4.0.0:
  Would remove:
    /opt/homebrew/bin/bidscoin
    /opt/homebrew/bin/bidscoiner
    /opt/homebrew/bin/bidseditor
    /opt/homebrew/bin/bidsmapper
    /opt/homebrew/bin/bidsparticipants
    /opt/homebrew/bin/deface
    /opt/homebrew/bin/dicomsort
    /opt/homebrew/bin/echocombine
    /opt/homebrew/bin/medeface
    /opt/homebrew/bin/physio2tsv
    /opt/homebrew/bin/plotphysio
    /opt/homebrew/bin/rawmapper
    /opt/homebrew/bin/skullstrip
    /opt/homebrew/bin/slicereport
    /opt/homebrew/lib/python3.11/site-packages/bidscoin-4.0.0-py3.11.egg
Proceed (Y/n)? Y
  Successfully uninstalled bidscoin-4.0.0

[marcelzwiers]

I think installing bidscoin-qt6 in a virtual environment, as you did, should just work. I don't own a mac, so I can't replicate your issue myself, but I'll try to find a colleague with a mac who can test this out for me. In the meantime, perhaps this is of help? https://forum.qt.io/topic/136786/how-to-fix-the-error-qtwidgets-abi3-so-reason-image-not-found-in-python-on-mac-os-x-catalina/2

[marcelzwiers]

Or this one... https://stackoverflow.com/questions/65901162/how-can-i-run-pyqt5-on-my-mac-with-m1chip-ppc64el-architecture

[dkp]

Thank you for the suggestions. I tried the regular bidscoin distribution on linux (intel) the other day and got errors there too (though completely different):

x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/dpat/projects/Testing/bidscoin/venv_bc/include -I/usr/include/python3.11 -c traits/ctraits.c -o build/temp.linux-x86_64-cpython-311/traits/ctraits.o
      error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for traits
Failed to build traits
ERROR: Could not build wheels for traits, which is required to install pyproject.toml-based projects

I'll probably try apptainer on our HPC next, but it may be a little while before I get to it. I'll let you know how it goes. I appreciate your willingness to help.

[marcelzwiers]

I have never seen this message and have no idea what could be the cause of it. I'll look into it tomorrow...

[marcelzwiers]

That traits error seems rather fundamental, can you install other packages without error? E.g. pip install pydicom?

[dkp]

Aha, thank you for your patience. Problem solved on Linux. There were two factors, I think:

• For conda, it is important that it be installed for the user (we had a system with conda installed "globally" and this then resulted in permission problems.
• Again, for conda, it is critical to explicitly specify the version of python when creating the environment: e.g., conda create -n bidscoin python=3.10

[marcelzwiers]

• For conda, it is important that it be installed for the user (we had a system with conda installed "globally" and this then resulted in permission problems.

You mean that you have to install it in a (e.g. conda) virtual environment? Sure, that is not just true for BIDScoin, but for any python library you install

• Again, for conda, it is critical to explicitly specify the version of python when creating the environment: e.g., conda create -n bidscoin python=3.10

Although I have seen this clunky conda behaviour too in the past, it is certainly not true in general (in other OS / anaconda versions it works just fine)

[marcelzwiers]

Which Linux version is that? Some old CentOS version by any chance?

[dkp]

This is ubuntu 22.0.2: I may misunderstand what I'm encountering, but I'm talking about who installs CONDA not who uses conda to create an environment. In the case of our local linux box, my husband set everything up. He installed conda in /usr/local/src. But it seemed that carried all sorts of permissions baggage with it such that my account had some, but not all, relevant permissions for setting up individual conda environments. Anyhow, part of the "fix" was to install miniconda for each user separately.

I've always set up conda for myself, so I was not aware of what issues might ensue from a global install.

[marcelzwiers]

Thanks, I didn't know that. Glad you figured that out :)

[dkp]

Well, maybe there is some way around the problem, I just don't know what that is yet.

[bendhouseart]

Just got back from summer break, so apologies for the late reply.

I think this library error is a bit out of my scope, @bendhouseart have you perhaps seen this?

Hi @marcelzwiers don't recall running into the specific errors in the attached text file. But, happy to test this out on Apple Silicon for you and @dkp

Currently getting hanging doing the install w/ dependencies for pyqt5 via python 3.10:

Collecting bidscoin
  Using cached bidscoin-4.0.0-py3-none-any.whl (655 kB)
Collecting pandas (from bidscoin)
  Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl (10.8 MB)
Collecting matplotlib (from bidscoin)
  Downloading matplotlib-3.7.2-cp310-cp310-macosx_11_0_arm64.whl (7.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 8.9 MB/s eta 0:00:00
Collecting numpy (from bidscoin)
  Downloading numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 17.7 MB/s eta 0:00:00
Collecting pydicom>=2 (from bidscoin)
  Downloading pydicom-2.4.1-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 20.5 MB/s eta 0:00:00
Collecting PyQt5>=5.12.1 (from bidscoin)
  Downloading PyQt5-5.15.9.tar.gz (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 21.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... -

That said, installing from here git clone --branch qt6 https://github.com/Donders-Institute/bidscoin.git, does reproduce the ruamel.yaml error.... so that's something.

Really frustrating when even the example in their docs fails miserably. But, that's to be expected I think when the docs are a year out of date at least:

>>> from ruamel.yaml import YAML
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'YAML' from 'ruamel.yaml' (/Users/galassiae/Projects/TestInstallBidscoinFromPIPversionQT6/bidscoin/venv/lib/python3.10/site-packages/ruamel.yaml.clib-0.2.7-py3.10-macosx-12-arm64.egg/ruamel/yaml/__init__.py)

If I roll back to rumel.yaml version 0.17.0 I no longer get that error:

(venv) galassiae@MH02276145MLI bidscoin % pip install ruamel.yaml==0.17.0
Collecting ruamel.yaml==0.17.0
  Downloading ruamel.yaml-0.17.0-py2.py3-none-any.whl (101 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.5/101.5 kB 1.9 MB/s eta 0:00:00
WARNING: The candidate selected for download or install is a yanked version: 'ruamel-yaml' candidate (version 0.17.0 at https://files.pythonhosted.org/packages/69/b8/ea30b2f7a38680be7c823428d502e009e2f2a9d7122e57221bf6397bc77f/ruamel.yaml-0.17.0-py2.py3-none-any.whl (from https://pypi.org/simple/ruamel-yaml/))
Reason for being yanked: would still install for Python 2
Installing collected packages: ruamel.yaml
Successfully installed ruamel.yaml-0.17.0

[notice] A new release of pip is available: 23.1.2 -> 23.2
[notice] To update, run: pip install --upgrade pip
(venv) galassiae@MH02276145MLI bidscoin % python
Python 3.10.11 (main, Apr  7 2023, 07:24:47) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ruamel.yaml import YAML
>>> 

It might be worth rolling back to an earlier version of this library or ditching it for pyYAML or something else.

@marcelzwiers I'm going to fiddle around with the qt6 branch mentioned earlier, let me know if you would rather update it from master/current release before I get too carried away.

[bendhouseart]

Alright, moving over discussion from mattermost to here so it's more visible.

Turns out I wasn't just "tired and out of it" pip install . has some real issues with python 3.11 and Apple. Seems to time out after about an hour of failing to resolve a new pyproject.toml/lock file, see:

  Using cached PyQt5-5.15.9.tar.gz (3.2 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... -

[marcelzwiers]

@bendhouseart Well, I also have trouble running tox with python 3.11, but that's because of pypet2bids depending on a specific python version, i.e. from pypet2bids/pyproject.toml:

[tool.poetry.dependencies]
python = ">=3.8,<=3.11.1"

It would make my life easier if you would relax it a bit and take out the point release (I think that's a harmless thing to do): python = ">=3.8,<=3.11"

Here's the error:

py311: install_package_deps> python -I -m pip install 'pypet2bids>=1.0.12'
ERROR: Ignored the following versions that require a different python version: 0.0.10 Requires-Python >3.7.1,<3.10; 0.0.11 Requires-Python >3.7.1,<3.10; 0.0.12 Requires-Python >3.7.1,<3.10; 0.0.13 Requires-Python >3.7.1,<3.10; 0.0.14 Requires-Python >3.7.1,<3.10; 0.0.15 Requires-Python >3.7.1,<3.10; 0.0.4 Requires-Python >3.7.1,<3.10; 0.0.6 Requires-Python >3.7.1,<3.10; 0.0.7 Requires-Python >3.7.1,<3.10; 0.0.8 Requires-Python >3.7.1,<3.10; 0.0.9 Requires-Python >3.7.1,<3.10; 1.0.0 Requires-Python >3.7.1,<3.10; 1.0.10 Requires-Python >=3.8,<3.11; 1.0.12 Requires-Python >=3.8,<=3.11.1; 1.0.2 Requires-Python >3.7.1,<3.10; 1.0.5 Requires-Python >=3.8,<3.11; 1.0.7 Requires-Python >=3.8,<3.11; 1.0.8 Requires-Python >=3.8,<3.11; 1.0.9 Requires-Python >=3.8,<3.11; 1.1.0 Requires-Python >=3.8,<=3.11.1; 1.1.1 Requires-Python >=3.8,<=3.11.1; 1.1.2 Requires-Python >=3.8,<=3.11.1; 1.2.2 Requires-Python >=3.8,<=3.11.1; 1.2.3 Requires-Python >=3.8,<=3.11.1
ERROR: Could not find a version that satisfies the requirement pypet2bids>=1.0.12 (from versions: 1.0.4)
ERROR: No matching distribution found for pypet2bids>=1.0.12

[marcelzwiers]

@bendhouseart I chose ruamel.yaml over pyyaml because pyyaml still doesn't support the 10 year old release of yaml v1.2. Generally, it has been very good to me, including supporting writing back the comments (which I think pyyaml just drops). Which version of ruamel.yaml wasn't working for you? I suspect it is the compiled part (see ruamel.yaml.clib-0.2.7-py3.10-macosx-12-arm64.egg) that is giving problems... Perhaps you can use conda-forge?

https://anaconda.org/conda-forge/ruamel.yaml

[marcelzwiers]

@bendhouseart Perhaps you can file a ruamel.yaml ticket? I think the author is very responsive to good tickets https://sourceforge.net/p/ruamel-yaml/tickets/

[marcelzwiers]

@bendhouseart I've been reading up a bit on how to use python on MacOS and learned that you probably don't want to use brew to create virtual environments, but use conda, venv or pyenv or so. I saw in the qt6 log that the yaml library was reading the yaml file just fine. Also, using setup.py is not always the same as using pip, so to be safe, I'd use the latter to make sure all dependencies are installed

[marcelzwiers]

Ok, I managed to borrow a brand new M2 MacBook pro (out of the box). I did the following

1. Installed a standard python 3.11.5 interpreter using the universal installer from here: https://www.python.org/downloads/macos/
2. I then installed SSL certificates for python by running the Install Certificates.command in the python application folder (this is not really needed but used for downloading tutorial data and checking the latest pypi version)
3. I created a standard virtual environment using: python3 -m venv bidscoin
4. I installed the qt6 branch using: source bidscoin/bin/activate; python3 -m pip install git+https://github.com/Donders-Institute/bidscoin@qt6
5. Running the test bidscoin -t just works normally (except for plugin errors, which I didn't install)
6. Runing the bidsmapper and bidseditor also work normally (with GUI)

So no ruamel.yaml error and no pyqt6 error. It seems to me the problems are caused by homebrew

[marcelzwiers]

I'm closing this issue because it is due to the user installation. Feel free to re-open it if needed