Installing native glue for M1/M2 (MacOSX-arm64)

jfoster17 commented 1 year ago

I'm adding some notes here about getting a native version of glue running natively on Mac M1/M2 Apple Silicon (MacOSX-arm64) machines. Doing this for basic glue is actually pretty simple using Mamba. Note that it's a pain to set things up so that you can have some environments using Intel (x86_64) and some environments using Apple Silicon, but it's certainly getting close to the point where people can just use native python, so if you're brave:

mamba create -n glue-native-arm64 python=3.10
mamba activate glue-native-arm64
mamba install glue-core

This gets me a functional glue version (1.6.0 today) with just the conda-forge channel configured. Unfortunately nothing provides glueviz for MacOSX-arm64 so this means I don't have vispy viewers.

mamba install glue-vispy-viewers

This gets me version 1.0.4 of glue-vispy-viewers and a pretty recent vispy (note that if you have glueviz channel configured you get a really old vispy here that does not work at all) but they don't work because of an error with echo. There doesn't seem to be a recent conda package of echo (on conda-forge or glueviz) -- the latest is 0.5 and this does not work with the latest glue-vispy-viewers.

So we do:

pip install echo --upgrade

Which gets us the recent echo but warns us that there are some missing dependencies for glue-core:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. glue-core 1.6.0 requires openpyxl>=3.0, which is not installed. glue-core 1.6.0 requires pvextractor>=0.2, which is not installed.

And the vispy viewers don't actually work (the scatter viewer is created, but does not display points).

❯ mamba list | grep vispy glue-vispy-viewers 1.0.4 pyhd8ed1ab_0 conda-forge vispy 0.12.1 py310hb614ae6_0 conda-forge

But if we use pip to upgrade:

pip install --upgrade glue-vispy-viewers

Now we get the latest version of glue-vispy-viewers (along with a bunch of other things like openpyxl and pvextractor that pip complained about before):

❯ mamba list | grep vispy glue-vispy-viewers 1.0.6 pypi_0 pypi vispy 0.12.1 py310hb614ae6_0 conda-forge

And everything seems to work.

So what have we learned?

You can use mamba and conda-forge to get a native version of glue running on Apple Silicon, but you also have to pip install some stuff because a bunch of packages don't have modern enough versions on conda-forge.
(I totally failed to get a native working QT + glue setup going without mamba/conda, which is why I went this route)
There's some odd problem with some dependencies missing from the conda-forge version of glue-core.

jfoster17 commented 1 year ago

@astrofrog -- as requested yesterday here are my notes on getting a native Apple Silicon version of glue working.

astrofrog commented 1 year ago

Thank you! Just curious, what happens if you make a mamba environment with just python and pip and then pip install glueviz into it?

jfoster17 commented 1 year ago

I had trouble getting QT set up that way.

So I get what looks like all the appropriate versions of python packages but then this error:

❯ glue -v Input arguments: ['/Users/jfoster/mambaforge/envs/just-pip/bin/glue', '-v'] Traceback (most recent call last): File "/Users/jfoster/mambaforge/envs/just-pip/lib/python3.10/site-packages/qtpy/init.py", line 252, in from PySide6 import version as PYSIDE_VERSION # analysis:ignore ModuleNotFoundError: No module named 'PySide6'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/jfoster/mambaforge/envs/just-pip/bin/glue", line 8, in sys.exit(main()) File "/Users/jfoster/mambaforge/envs/just-pip/lib/python3.10/site-packages/glue/main.py", line 253, in main start_glue(**kwargs) File "/Users/jfoster/mambaforge/envs/just-pip/lib/python3.10/site-packages/glue/main.py", line 141, in start_glue from qtpy import QtWebEngineWidgets # noqa File "/Users/jfoster/mambaforge/envs/just-pip/lib/python3.10/site-packages/qtpy/init.py", line 259, in raise QtBindingsNotFoundError() qtpy.QtBindingsNotFoundError: No Qt bindings could be found

glueviz doesn't specify a qt extra...

If I specifically do (e.g.) pip install PyQt5 or pip install "glue-core[qt]" then I get an error because I don't have qmake. I think I had some trouble getting qmake working when I originally tried to go this route but I don't recall specifically.

dhomeier commented 1 year ago

It's not installing the backend automatically, and it seems glueviz does not really support specific envs - I can run e.g. pip install glueviz[qt] or glueviz[test], but it does not seem to doe anything, while pip install glue-core[test] will do the expected thing, and pip install glue-core[qt] will try to install PyQt5 as default backend (with no backend installed, it somehow ends up lost with PySide6 as the last option tried). But on Ventura/arm64 as I mentioned elsewhere, PyQt5 seems not really well supported, so one useful path I found would be

pip install glueviz
pip install pyqt6
pip install glue-core[test]  # if you want to run pytest

That should also bring you glue-core==1.7.0 and up-to-date versions of the other packages.

First installing everything you can with mamba works as well (I tend to trust the compiled extensions from mamba/conda more than the pip ones), but you'll need to explicitly upgrade some packages e.g. to get from glue-core 1.6.0 to 1.7.0. Dunno why condaforge is so slow in updating and does not seem to catch some packages like echo at all; that's been a regular issue in support questions here. And mamba install glueviz will install a backend for you, but always default to qt5, so you will also have to do an extra pip install qyqt6 (and possibly pip uninstall pyqt5 pyqt5-sip so glue won't detect qt5 first) if you want a different backend.

dhomeier commented 1 year ago

On the topic of dealing with some stuff that may not yet run natively, I have modified my conda setup such that I could create two separate installations for native arm64 and x86_64 using Rosetta and have it detect the current arch:

if [ "$(uname -m)" = x86_64 ]; then
    MAMBA=${MAMBA:-mambax86}
else
    MAMBA=${MAMBA:-mambaforge}
fi
print "Initialising Mambaforge in ~/opt/$MAMBA ..."
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$(~/opt/${MAMBA}/bin/conda 'shell.zsh' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "${HOME}/opt/${MAMBA}/etc/profile.d/conda.sh" ]; then
        . "${HOME}/opt/${MAMBA}/etc/profile.d/conda.sh"
    else
        export PATH="${HOME}/opt/${MAMBA}/bin:$PATH"
    fi
fi
unset __conda_setup

if [ -f "${HOME}/opt/${MAMBA}/etc/profile.d/mamba.sh" ]; then
    . "${HOME}/opt/${MAMBA}/etc/profile.d/mamba.sh"
fi
# <<< conda initialize <<<

If you open a Terminal window/tab with e.g. arch -x86_64 zsh as shell, this will launch the (Rosetta-installed) mamba environment in ~/opt/mambax86; but that's certainly going some way into the experimental (and you probably should not re-run mamba init or it will wipe out these customisations).

dhomeier commented 1 year ago

Thank you! Just curious, what happens if you make a mamba environment with just python and pip and then pip install glueviz into it?

On yet another note, on Monterey and Ventura you can even get a functional glue environment without installing any extra Python just using the system /usr/bin/python3, pip3 install glueviz; pip3 install pyqt6. But that of course does not give you any virtual env capability, and running on Apple's Python 3.9.6, not updated since probably macOS 12.1, I'd not really recommend that option.

jfoster17 commented 1 year ago

On the topic of dealing with some stuff that may not yet run natively, I have modified my conda setup such that I could create two separate installations for native arm64 and x86_64 using Rosetta and have it detect the current arch:

Thanks, this is helpful!

astrofrog commented 1 year ago

I just checked my set-up - what I have done on my M1 mac is installed Python 3.11 from python.org and then I create virtual environments with e.g.

python -m venv ~/python/dev

then I install glue with:

source ~/python/dev/bin/activate
pip install glueviz PyQt6

(well in practice I install glue from the repos but that's how I'd get a functional native install on M1)

dhomeier commented 1 year ago

That might be the cleanest path, as condaforge still has no PyQt6 for arm64, so there's some crucial components that need to be pip-installed anyway. Probably depends on whether one prefers the conda/mamba venv system.

glue-viz / glue

Installing native glue for M1/M2 (MacOSX-arm64) #2359