Open dokato opened 2 years ago
@da-wad @hmaarrfk Can you please comment on current status of zfp pip and conda packages, particularly with respect to M1?
can you please provide the output of:
pip list
conda list
in your virtual environment
If you are using conda, providing
conda info
also really helps.
As for the status on conda-forge, we have zfpy compiled and built. However, we have no way to test it on any machine.
Hi, unfortunately there is still no pip package for zfpy as Github Actions does not provide M1 silicon to build on. While cross-compilation to arm64 on x86 seems to work, there is an issue with the subsequent linking/delocalization which we haven't managed to solve. If conda packages aren't working either, users of M1 need to do their own compilation at the moment.
@hmaarrfk sure:
$ pip list
Package Version
----------------------------- -------
appnope 0.1.3
asttokens 2.0.8
backcall 0.2.0
backports.functools-lru-cache 1.6.4
decorator 5.1.1
executing 1.1.1
ipython 8.5.0
jedi 0.18.1
matplotlib-inline 0.1.6
numpy 1.23.3
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pip 22.2.2
prompt-toolkit 3.0.31
ptyprocess 0.7.0
pure-eval 0.2.2
Pygments 2.13.0
setuptools 65.4.1
six 1.16.0
stack-data 0.5.1
traitlets 5.4.0
wcwidth 0.2.5
wheel 0.37.1
$ conda list
# Name Version Build Channel
appnope 0.1.3 pyhd8ed1ab_0 conda-forge
asttokens 2.0.8 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bzip2 1.0.8 h3422bc3_4 conda-forge
ca-certificates 2022.9.24 h4653dfc_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
executing 1.1.1 pyhd8ed1ab_0 conda-forge
ipython 8.5.0 pyhd1c38e8_1 conda-forge
jedi 0.18.1 pyhd8ed1ab_2 conda-forge
libblas 3.9.0 16_osxarm64_openblas conda-forge
libcblas 3.9.0 16_osxarm64_openblas conda-forge
libcxx 14.0.6 h2692d47_0 conda-forge
libffi 3.4.2 h3422bc3_5 conda-forge
libgfortran 5.0.0 11_3_0_hd922786_25 conda-forge
libgfortran5 11.3.0 hdaf2cc0_25 conda-forge
liblapack 3.9.0 16_osxarm64_openblas conda-forge
libopenblas 0.3.21 openmp_hc731615_3 conda-forge
libsqlite 3.39.4 h76d750c_0 conda-forge
libzlib 1.2.12 h03a7124_4 conda-forge
llvm-openmp 14.0.4 hd125106_0 conda-forge
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
ncurses 6.3 h07bb92c_1 conda-forge
numpy 1.23.3 py310h5d7c261_0 conda-forge
openssl 3.0.5 h03a7124_2 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pip 22.2.2 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.31 pyha770c72_0 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
pygments 2.13.0 pyhd8ed1ab_0 conda-forge
python 3.10.6 hae75cb6_0_cpython conda-forge
python_abi 3.10 2_cp310 conda-forge
readline 8.1.2 h46ed386_0 conda-forge
setuptools 65.4.1 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
stack_data 0.5.1 pyhd8ed1ab_0 conda-forge
tk 8.6.12 he1e0b03_0 conda-forge
traitlets 5.4.0 pyhd8ed1ab_0 conda-forge
tzdata 2022e h191b570_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h57fd34a_0 conda-forge
zfp 1.0.0 h7b19444_1 conda-forge
zfpy 1.0.0 py310h447c399_1 conda-forge
$ conda info
active environment : tstzfpy
active env location : /Users/dd/anaconda3/envs/tstzfpy
shell level : 2
user config file : /Users/dd/.condarc
populated config files : /Users/dd/miniforge3/.condarc
/Users/dd/.condarc
conda version : 22.9.0
conda-build version : not installed
python version : 3.9.13.final.0
virtual packages : __osx=12.6=0
__unix=0=0
__archspec=1=arm64
base environment : /Users/dd/miniforge3 (writable)
conda av data dir : /Users/dd/miniforge3/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
https://conda.anaconda.org/conda-forge/noarch
package cache : /Users/dd/miniforge3/pkgs
/Users/dd/.conda/pkgs
envs directories : /Users/dd/anaconda3/envs
/Users/dd/Library/r-miniconda
/Users/dd/Library/r-miniconda/envs
/Users/dd/miniforge3/envs
/Users/dd/.conda/envs
platform : osx-arm64
user-agent : conda/22.9.0 requests/2.28.1 CPython/3.9.13 Darwin/21.6.0 OSX/12.6
UID:GID : 501:20
netrc file : /Users/dd/.netrc
offline mode : False
I can't spot anything immediately wrong with your environment.
It is quite possible that you are the first testing the package on conda forge.
It is quite possible that we have a bug in it.
I have exactly the same problems on a M1 Mac. If it is helpful, I can also show some more information about my environment.
Meanwhile, I had a lot of problems building from scratch. I want to leave a step-by-step guide here so someone else who sees the problem has an easier time. I'm neither proficient with C-extensions nor CMake, so apologies if this is very hacky.
First, clone ZFP repository. Then, do the following (for Python 3.10, change for other python versions accordingly):
mkdir build
cd build
cmake .. -DBUILD_ZFPY=ON \
-DCMAKE_BUILD_TYPE=RELEASE \
-DPYTHON_EXECUTABLE:FILEPATH="/opt/homebrew/bin/python3.10" \
-DPYTHON_LIBRARY="/opt/homebrew/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib" \
-DPYTHON_INCLUDE_DIR="/opt/homebrew/Frameworks/Python.framework/Versions/3.10/include/python3.10"
Under bin
you now find a .so
library, which you can import with
import zfpy
if you started your python interpreter in the corresponding directory.
To make the package importable from anywhere, first go back to the project root.
Then we change the setup.py
file: find and remove the library_dirs
attribute completely. Next,
install the package with:
pip install .
Lastly, we manually copy the libs
from the build process to a lib path on our system. This might look something like this:
cp build/lib/* /opt/homebrew/Caskroom/mambaforge/base/envs/climate/lib/
If you have no clue where to place the lib, just open up python
and try to
import zfpy
Python will complain that it didn't find the library and will show you the content of your rpath
.
Hi! I have the same issue as @dokato on my M1 mac. Thanks!
See https://github.com/seung-lab/cloud-volume/issues/561 and https://github.com/seung-lab/cloud-volume/issues/570 for some workarounds @YijieYin
I've been skimming through https://github.com/LLNL/zfp/issues/175#issue-1338357172
I'm rebuilding the conda-forge package with the OSX_ARCHITECTURES flags specified https://github.com/conda-forge/zfpy-feedstock/pull/32
If somebody is still interested in conda-forge packages, they can follow the progress there. Other ideas are welcome too.
I tried multiple times to install via pip, or conda. Pip gives some missing dll library error, so then I discovered this thread on wheels so I gave a go with conda: https://github.com/LLNL/zfp/issues/175. The installation goes flawless, but then importing the package crashes with segmentation fault:
Details of my system:
I used a clean conda environment for that, just jupyter notebook and numpy installed really.