search

xraypy / xraylarch

Larch: Applications and Python Library for Data Analysis of X-ray Absorption Spectroscopy (XAS, XANES, XAFS, EXAFS), X-ray Fluorescence (XRF) Spectroscopy and Imaging, and more.
https://xraypy.github.io/xraylarch
Other
127 stars 62 forks source link

bug during install on a M1 Mac with System ist 13.3.1 #430

Closed StefanMangoldKa closed 1 year ago

StefanMangoldKa commented 1 year ago

Collecting fabio Using cached fabio-2023.4.1.tar.gz (724 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error

Happens during Script and Anaconda install !!

StefanMangoldKa commented 1 year ago

Collecting fabio Using cached fabio-2023.4.1.tar.gz (724 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1

newville commented 1 year ago

@StefanMangoldKa Hm, I'm not sure what is going on there.

I recommend using the GetLarch.sh script -- it should be more up-to-date than the package installer and also gives better error messages into GetLarch.log. If you try that, maybe the log will better describe the error.

FWIW, the GetLarch.sh install script should install an x86_64 binary set, as it seems that many codes are not yet fully ported to ARM M1. But it all seems to work for me on a Macbook with M1 Max.

StefanMangoldKa commented 1 year ago

I used the script just downloaded today from the website and got the error

Collecting fabio Using cached fabio-2023.4.1.tar.gz (724 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [17 lines of output]

StefanMangoldKa commented 1 year ago

log-File does not exists! Tried to access it, but not there ....

StefanMangoldKa commented 1 year ago

already pip-install-ohyzynpz is not existing ....

StefanMangoldKa commented 1 year ago

I have a second Mac with M1. maybe I try to install it there also. The install worked on a old MacPro (but with an older OS)

newville commented 1 year ago

@StefanMangoldKa Please slow down, and try things carefully, and give complete reports.

Open a terminal and move to the folder with the GetLarch.sh. In that terminal, run 

rm -rf ~/xraylarch
rm -rf ~/Library/Caches/pip/wheels
sh GetLarch.sh

I just did this, and the installation worked for me.

Running GetLarch.sh WILL DEFINITELY generate a log file named GetLarch.log in the same folder as the GetLarch.sh script. If you have problems, send that log file. The whole thing.

I doubt very much that the problem is related to being M1 or macOS 13.3.

StefanMangoldKa commented 1 year ago

same error with doing this

rm -rf ~/xraylarch rm -rf ~/Library/Caches/pip/wheels sh GetLarch.sh

(GetLarch.sh) was in the Download directory

Don't see the log file in the Download directory, What is the name ....

StefanMangoldKa commented 1 year ago

got the name ´, but the file is old, so I will redo the stuff ..., and before I will delete the file.

StefanMangoldKa commented 1 year ago

##############

This script will install Larch for MacOSX to /Users/stefanmangold/xraylarch

The following packages will be taken from conda-forge:

numpy=>1.22 scipy=>1.8 matplotlib=>3.6 scikit-image scikit-learn pycifrw pandas jupyter plotly wxpython

See GetLarch.log for complete log and error messages

##############

Installing Miniconda for MacOSX to /Users/stefanmangold/xraylarch

> sh ./Mambaforge-MacOSX-x86_64.sh -b -p /Users/stefanmangold/xraylarch

PREFIX=/Users/stefanmangold/xraylarch Unpacking payload ... Extracting bzip2-1.0.8-h0d85af4_4.tar.bz2 Extracting c-ares-1.18.1-h0d85af4_0.tar.bz2 Extracting ca-certificates-2022.12.7-h033912b_0.conda Extracting icu-72.1-h7336db1_0.conda Extracting libcxx-16.0.0-h71dddab_0.conda Extracting libev-4.33-haf1e3a3_1.tar.bz2 Extracting libffi-3.4.2-h0d85af4_5.tar.bz2 Extracting libiconv-1.17-hac89ed1_0.tar.bz2 Extracting libzlib-1.2.13-hfd90126_4.tar.bz2 Extracting lzo-2.10-haf1e3a3_1000.tar.bz2 Extracting ncurses-6.3-h96cf925_1.tar.bz2 Extracting pybind11-abi-4-hd8ed1ab_3.tar.bz2 Extracting python_abi-3.10-3_cp310.conda Extracting reproc-14.2.4-hb7f2c08_0.conda Extracting tzdata-2023c-h71feb2d_0.conda Extracting xz-5.2.6-h775f41a_0.tar.bz2 Extracting fmt-9.1.0-hb8565cd_0.tar.bz2 Extracting libedit-3.1.20191231-h0678c8f_2.tar.bz2 Extracting libsolv-0.7.23-hbc0c0cd_0.conda Extracting libsqlite-3.40.0-ha978bb4_0.tar.bz2 Extracting libxml2-2.10.3-h554bb67_6.conda Extracting lz4-c-1.9.4-hf0c8a7f_0.conda Extracting openssl-3.1.0-hfd90126_0.conda Extracting readline-8.2-h9e318b2_1.conda Extracting reproc-cpp-14.2.4-hf0c8a7f_0.conda Extracting tk-8.6.12-h5dbffcc_0.tar.bz2 Extracting yaml-cpp-0.7.0-hf0c8a7f_2.tar.bz2 Extracting zstd-1.5.2-hbc0c0cd_6.conda Extracting krb5-1.20.1-h049b76e_0.conda Extracting libarchive-3.6.2-h6d8d9f1_0.conda Extracting libnghttp2-1.52.0-he2ab024_0.conda Extracting libssh2-1.10.0-h47af595_3.tar.bz2 Extracting python-3.10.10-he7542f4_0_cpython.conda Extracting certifi-2022.12.7-pyhd8ed1ab_0.conda Extracting charset-normalizer-3.1.0-pyhd8ed1ab_0.conda Extracting colorama-0.4.6-pyhd8ed1ab_0.tar.bz2 Extracting idna-3.4-pyhd8ed1ab_0.tar.bz2 Extracting libcurl-7.88.1-h6df9250_1.conda Extracting pluggy-1.0.0-pyhd8ed1ab_5.tar.bz2 Extracting pycosat-0.6.4-py310h90acd4f_1.tar.bz2 Extracting pycparser-2.21-pyhd8ed1ab_0.tar.bz2 Extracting pysocks-1.7.1-pyha2e5f31_6.tar.bz2 Extracting ruamel.yaml.clib-0.2.7-py310h90acd4f_1.conda Extracting setuptools-65.6.3-pyhd8ed1ab_0.conda Extracting toolz-0.12.0-pyhd8ed1ab_0.tar.bz2 Extracting wheel-0.40.0-pyhd8ed1ab_0.conda Extracting cffi-1.15.1-py310ha78151a_3.conda Extracting libmamba-1.4.1-h9d281b0_0.conda Extracting pip-23.0.1-pyhd8ed1ab_0.conda Extracting ruamel.yaml-0.17.21-py310h90acd4f_3.conda Extracting tqdm-4.65.0-pyhd8ed1ab_1.conda Extracting brotlipy-0.7.0-py310h90acd4f_1005.tar.bz2 Extracting cryptography-40.0.1-py310hdd0c95c_0.conda Extracting libmambapy-1.4.1-py310hb15139c_0.conda Extracting zstandard-0.19.0-py310h3cf44b0_1.conda Extracting conda-package-streaming-0.7.0-pyhd8ed1ab_1.conda Extracting pyopenssl-23.1.1-pyhd8ed1ab_0.conda Extracting conda-package-handling-2.0.2-pyh38be061_0.conda Extracting urllib3-1.26.15-pyhd8ed1ab_0.conda Extracting requests-2.28.2-pyhd8ed1ab_1.conda Extracting conda-23.1.0-py310h2ec42d9_0.conda Extracting mamba-1.4.1-py310h6bde348_0.conda

Installing base environment...

                                       __
      __  ______ ___  ____ _____ ___  / /_  ____ _
     / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
    / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
   / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
  /_/

Transaction

Prefix: /Users/stefanmangold/xraylarch

Updating specs:

───────────────────────────────────────────────────────────────────────────────────────

Transaction starting Linking bzip2-1.0.8-h0d85af4_4 Linking c-ares-1.18.1-h0d85af4_0 Linking ca-certificates-2022.12.7-h033912b_0 Linking icu-72.1-h7336db1_0 Linking libcxx-16.0.0-h71dddab_0 Linking libev-4.33-haf1e3a3_1 Linking libffi-3.4.2-h0d85af4_5 Linking libiconv-1.17-hac89ed1_0 Linking libzlib-1.2.13-hfd90126_4 Linking lzo-2.10-haf1e3a3_1000 Linking ncurses-6.3-h96cf925_1 Linking pybind11-abi-4-hd8ed1ab_3 Linking python_abi-3.10-3_cp310 Linking reproc-14.2.4-hb7f2c08_0 Linking tzdata-2023c-h71feb2d_0 Linking xz-5.2.6-h775f41a_0 Linking fmt-9.1.0-hb8565cd_0 Linking libedit-3.1.20191231-h0678c8f_2 Linking libsolv-0.7.23-hbc0c0cd_0 Linking libsqlite-3.40.0-ha978bb4_0 Linking libxml2-2.10.3-h554bb67_6 Linking lz4-c-1.9.4-hf0c8a7f_0 Linking openssl-3.1.0-hfd90126_0 Linking readline-8.2-h9e318b2_1 Linking reproc-cpp-14.2.4-hf0c8a7f_0 Linking tk-8.6.12-h5dbffcc_0 Linking yaml-cpp-0.7.0-hf0c8a7f_2 Linking zstd-1.5.2-hbc0c0cd_6 Linking krb5-1.20.1-h049b76e_0 Linking libarchive-3.6.2-h6d8d9f1_0 Linking libnghttp2-1.52.0-he2ab024_0 Linking libssh2-1.10.0-h47af595_3 Linking python-3.10.10-he7542f4_0_cpython Linking certifi-2022.12.7-pyhd8ed1ab_0 Linking charset-normalizer-3.1.0-pyhd8ed1ab_0 Linking colorama-0.4.6-pyhd8ed1ab_0 Linking idna-3.4-pyhd8ed1ab_0 Linking libcurl-7.88.1-h6df9250_1 Linking pluggy-1.0.0-pyhd8ed1ab_5 Linking pycosat-0.6.4-py310h90acd4f_1 Linking pycparser-2.21-pyhd8ed1ab_0 Linking pysocks-1.7.1-pyha2e5f31_6 Linking ruamel.yaml.clib-0.2.7-py310h90acd4f_1 Linking setuptools-65.6.3-pyhd8ed1ab_0 Linking toolz-0.12.0-pyhd8ed1ab_0 Linking wheel-0.40.0-pyhd8ed1ab_0 Linking cffi-1.15.1-py310ha78151a_3 Linking libmamba-1.4.1-h9d281b0_0 Linking pip-23.0.1-pyhd8ed1ab_0 Linking ruamel.yaml-0.17.21-py310h90acd4f_3 Linking tqdm-4.65.0-pyhd8ed1ab_1 Linking brotlipy-0.7.0-py310h90acd4f_1005 Linking cryptography-40.0.1-py310hdd0c95c_0 Linking libmambapy-1.4.1-py310hb15139c_0 Linking zstandard-0.19.0-py310h3cf44b0_1 Linking conda-package-streaming-0.7.0-pyhd8ed1ab_1 Linking pyopenssl-23.1.1-pyhd8ed1ab_0 Linking conda-package-handling-2.0.2-pyh38be061_0 Linking urllib3-1.26.15-pyhd8ed1ab_0 Linking requests-2.28.2-pyhd8ed1ab_1 Linking conda-23.1.0-py310h2ec42d9_0 Linking mamba-1.4.1-py310h6bde348_0 Transaction finished installation finished.

Installing packages from conda-forge

> /Users/stefanmangold/xraylarch/bin/mamba install -yc conda-forge numpy=>1.22 scipy=>1.8 matplotlib=>3.6 scikit-image scikit-learn pycifrw pandas jupyter plotly wxpython

Installing dependencies from PyPI

> /Users/stefanmangold/xraylarch/bin/pip install wxmplot wxutils lmfit asteval pyshortcuts pyfai

Collecting wxmplot Using cached wxmplot-0.9.55-py3-none-any.whl (95 kB) Collecting wxutils Using cached wxutils-0.3.0-py3-none-any.whl (40 kB) Collecting lmfit Using cached lmfit-1.2.1-py3-none-any.whl (101 kB) Collecting asteval Using cached asteval-0.9.29-py3-none-any.whl (18 kB) Collecting pyshortcuts Using cached pyshortcuts-1.9.0-py3-none-any.whl (158 kB) Collecting pyfai Using cached pyfai-2023.3.0-cp310-cp310-macosx_10_9_x86_64.whl (5.3 MB) Requirement already satisfied: numpy>=1.18 in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from wxmplot) (1.24.3) Requirement already satisfied: matplotlib>=3.5.0 in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from wxmplot) (3.7.1) Requirement already satisfied: wxPython>=4.1.0 in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from wxmplot) (4.2.0) Requirement already satisfied: Pillow>=7.0 in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from wxmplot) (9.5.0) Requirement already satisfied: pyyaml>=5.0 in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from wxmplot) (6.0) Requirement already satisfied: scipy>=1.6 in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from lmfit) (1.10.1) Collecting uncertainties>=3.1.4 Using cached uncertainties-3.1.7-py2.py3-none-any.whl (98 kB) Collecting numexpr Using cached numexpr-2.8.4-cp310-cp310-macosx_10_9_x86_64.whl (99 kB) Collecting h5py Using cached h5py-3.8.0-cp310-cp310-macosx_10_9_x86_64.whl (3.2 MB) Collecting silx Using cached silx-1.1.2-cp310-cp310-macosx_10_9_universal2.whl (6.7 MB) Collecting fabio Using cached fabio-2023.4.1.tar.gz (724 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error'

Installing xraylarch as 'pip install xraylarch'

> /Users/stefanmangold/xraylarch/bin/pip install xraylarch

Collecting xraylarch Using cached xraylarch-0.9.68-py3-none-any.whl (21.1 MB) Requirement already satisfied: matplotlib>=3.5 in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from xraylarch) (3.7.1) Requirement already satisfied: scikit-image in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from xraylarch) (0.20.0) Collecting uncertainties>=3.1 Using cached uncertainties-3.1.7-py2.py3-none-any.whl (98 kB) Collecting lmfit>=1.2.0 Using cached lmfit-1.2.1-py3-none-any.whl (101 kB) Requirement already satisfied: requests in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from xraylarch) (2.28.2) Requirement already satisfied: imageio in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from xraylarch) (2.28.1) Collecting sqlalchemy-utils Using cached SQLAlchemy_Utils-0.41.1-py3-none-any.whl (92 kB) Collecting xraydb>=4.5 Using cached xraydb-4.5.0-py3-none-any.whl (3.9 MB) Collecting h5py>=3.2 Using cached h5py-3.8.0-cp310-cp310-macosx_10_9_x86_64.whl (3.2 MB) Requirement already satisfied: charset-normalizer in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from xraylarch) (3.1.0) Requirement already satisfied: scikit-learn in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from xraylarch) (1.2.2) Requirement already satisfied: psutil in /Users/stefanmangold/xraylarch/lib/python3.10/site-packages (from xraylarch) (5.9.5) Collecting peakutils Using cached PeakUtils-1.3.4-py3-none-any.whl (7.7 kB) Collecting pymatgen Using cached pymatgen-2023.5.10-cp310-cp310-macosx_10_9_x86_64.whl (10.2 MB) Collecting pyfai Using cached pyfai-2023.3.0-cp310-cp310-macosx_10_9_x86_64.whl (5.3 MB) Collecting sqlalchemy>=2.0 Using cached SQLAlchemy-2.0.13-cp310-cp310-macosx_10_9_x86_64.whl (2.0 MB) Collecting termcolor Using cached termcolor-2.3.0-py3-none-any.whl (6.9 kB) Collecting fabio Using cached fabio-2023.4.1.tar.gz (724 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error' ##############

Larch Installation to /Users/stefanmangold/xraylarch done.

Applications can be run from the Larch folder on your Desktop.

To use from a terminal, you may want to add:

    export PATH=/Users/stefanmangold/xraylarch/bin:$PATH

to your /bin/bash startup script.

See GetLarch.log for complete log and error messages

##############

newville commented 1 year ago

@StefanMangoldKa Hm, there must still be some cached version of that "fabio-2023.4.1.tar.gz" file somewhere that is being used. Maybe somewhere under ~/.conda? You might need to search for that file fabio-2023.4.1.tar.gz and remove it (and maybe whatever folder it is in).

But, it also looks like you might have gotten all the way to a working Python/pip system with most of the packages installed. I'm not certain, but if that is the case you would not need to start from the beginning, but should be able to do from a Terminal:

~/xraylarch/bin/pip install fabio
~/xraylarch/bin/pip install xraylarch
StefanMangoldKa commented 1 year ago

~/xraylarch/bin/pip install fabio resulted in the same error message I deleted all, searched the mac "fabio-2023.4.1" and found noting .... deleted also some left overs from old anaconda installs in opt/ .... tried to reinstall again with exactly the same error. Maybe I try this afternoon a computer (iMac M1), which never had larch or anaconda installed.

StefanMangoldKa commented 1 year ago

found the reason:

If this is the case, the install via script works also on ARM Macs ….; the Intel-Mac, which I tested, had an updated version installed.

maurov commented 1 year ago

@newville my understanding of this issue is that fabio was already conda-installed when pip tried reinstalling it. This usually brings issues. We should avoid pip installing a dependency already installed via conda!

Unfortunately, I have already often seen errors like this on other Larch dependencies too, especially when running an update directly from xas_viewer. The solution is to clean both conda and pip installs and reinstall.

Since we cannot get out of conda due to Wxpython (and maybe others), I think having a conda-forge package for larch will solve most of the installation/update issues.

StefanMangoldKa commented 1 year ago

There were in principle 2.5 issues:

issue one: Update via update dialog in the XAS_Viewer destroyed the working installation on 1 Windows, 1 Intel-Mac and 1 ARM Mac.

issue 2: Install via binary worked only on the Intel Mac, I was not able to install via Script or binary Larch on the MacbookAir. Since Matt claimed to be able to make the install, I tested the install on a clean iMac M1. during the install via script I got the same broken install as before, but the message that the command line tool of the developer tools are needed because of one command. Did the install and then the install via script worked. Updated the developer tool on the MacbookAir and the install worked. Seen that intel mac had a complete install of an updated developer tools including command line tools installed.

-> so core problem -> one needs updated dev tools and the command line tool (not standard) on the machine .... and you get only an error message from the MacOS, if nothing is installed.

Do one really need this command? If yes, you should warn the people, that they need this ...

newville commented 1 year ago

@maurov I think the problem here might be that we are not very specific about when/where to install fabio, and that it is installing only because of pyfai, which was specified to install from pypi (and does install, as a wheel). While there are wheels for fabio on PyPI, those were not being used, so it was installing from source and compiled locally on macOS, which then requires the Xcode command-line tools. That can work, but it is definitely not the intention.

In general, pip will recognize that there was a "current enough" package installed by conda and then know it does not need to install that explicitly by pip (like, matplotlib is a dependency for lots of things, but doesn't get reinstalled from pip).

So, I sort of think this is a weirdness with pyfai/fabio/pypi - maybe how the fabio dependency is defined. Installing fabio (and maybe pyfai) from conda-forge seems a perfectly reasonable solution. We could just increase the number of packages taken from conda-forge (and, as with #433, be explicit about always grabbing that wxPython) in GetLarch.sh. It is always a bit a question about how hard of a dependency wxPython should be for Larch. I'm willing to say that all installation scripts that install into a conda-based environment should be explicit about including wxPython, and people wanting to install from source or into other Python environments (mostly, Linux), can deal with installing wxPython.

My preference is definitely to not use conda for xraylarch itself. Conda turns out to be "very clever" and notices and tries to fix things that it should not touch, especially with pre-compiled binaries. I think we would have to split up the "binaries" (Feff, especially) and the Python code. That would be some real work.

For "needs conda at all", wxPython on Linux is the only hard requirement that cannot be found on PyPi (though fabio on PyPI seems to not work). That said, I have no problem with using conda/conda-forge for lots of the "scipy stack". I think we have seen only a couple problems with mixing pypi and conda. We could put this issue in that category, though I suspect it is really some oddity with how the pypi packages are defined (and that a PyPI-only install would also have this problem).

@StefanMangoldKa The Windows issue is separate (see #352) from the macOS issues. I think there will be no difference between macOS ARM and x86_64 (and see no evidence that there might be).

On macOS, the current GetLarch.sh does appear to be downloading fabio as source and trying to install, which needs the compiler stack. That is not what we intend, and is fixable.