Segmentation fault on Mac M1

[dokato]

I tried multiple times to install via pip, or conda. Pip gives some missing dll library error, so then I discovered this thread on wheels so I gave a go with conda: https://github.com/LLNL/zfp/issues/175. The installation goes flawless, but then importing the package crashes with segmentation fault:

$ ipython
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:38:29) [Clang 13.0.1 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import zfpy
[1]    3650 segmentation fault  ipython

Details of my system:

$ arch
arm64
$ system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 12.6 (21G115)
      Kernel Version: Darwin 21.6.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal

I used a clean conda environment for that, just jupyter notebook and numpy installed really.

[lindstro]

@da-wad @hmaarrfk Can you please comment on current status of zfp pip and conda packages, particularly with respect to M1?

[hmaarrfk]

can you please provide the output of:

pip list

conda list

in your virtual environment

[hmaarrfk]

If you are using conda, providing

conda info

also really helps.

As for the status on conda-forge, we have zfpy compiled and built. However, we have no way to test it on any machine.

[da-wad]

Hi, unfortunately there is still no pip package for zfpy as Github Actions does not provide M1 silicon to build on. While cross-compilation to arm64 on x86 seems to work, there is an issue with the subsequent linking/delocalization which we haven't managed to solve. If conda packages aren't working either, users of M1 need to do their own compilation at the moment.

[dokato]

@hmaarrfk sure:

$ pip list
Package                       Version
----------------------------- -------
appnope                       0.1.3
asttokens                     2.0.8
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
decorator                     5.1.1
executing                     1.1.1
ipython                       8.5.0
jedi                          0.18.1
matplotlib-inline             0.1.6
numpy                         1.23.3
parso                         0.8.3
pexpect                       4.8.0
pickleshare                   0.7.5
pip                           22.2.2
prompt-toolkit                3.0.31
ptyprocess                    0.7.0
pure-eval                     0.2.2
Pygments                      2.13.0
setuptools                    65.4.1
six                           1.16.0
stack-data                    0.5.1
traitlets                     5.4.0
wcwidth                       0.2.5
wheel                         0.37.1

$ conda list
# Name                    Version                   Build  Channel
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
asttokens                 2.0.8              pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2022.9.24            h4653dfc_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
executing                 1.1.1              pyhd8ed1ab_0    conda-forge
ipython                   8.5.0              pyhd1c38e8_1    conda-forge
jedi                      0.18.1             pyhd8ed1ab_2    conda-forge
libblas                   3.9.0           16_osxarm64_openblas    conda-forge
libcblas                  3.9.0           16_osxarm64_openblas    conda-forge
libcxx                    14.0.6               h2692d47_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           11_3_0_hd922786_25    conda-forge
libgfortran5              11.3.0              hdaf2cc0_25    conda-forge
liblapack                 3.9.0           16_osxarm64_openblas    conda-forge
libopenblas               0.3.21          openmp_hc731615_3    conda-forge
libsqlite                 3.39.4               h76d750c_0    conda-forge
libzlib                   1.2.12               h03a7124_4    conda-forge
llvm-openmp               14.0.4               hd125106_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h07bb92c_1    conda-forge
numpy                     1.23.3          py310h5d7c261_0    conda-forge
openssl                   3.0.5                h03a7124_2    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.31             pyha770c72_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
python                    3.10.6          hae75cb6_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1.2                h46ed386_0    conda-forge
setuptools                65.4.1             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
stack_data                0.5.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
traitlets                 5.4.0              pyhd8ed1ab_0    conda-forge
tzdata                    2022e                h191b570_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
zfp                       1.0.0                h7b19444_1    conda-forge
zfpy                      1.0.0           py310h447c399_1    conda-forge

$ conda info
     active environment : tstzfpy
    active env location : /Users/dd/anaconda3/envs/tstzfpy
            shell level : 2
       user config file : /Users/dd/.condarc
 populated config files : /Users/dd/miniforge3/.condarc
                          /Users/dd/.condarc
          conda version : 22.9.0
    conda-build version : not installed
         python version : 3.9.13.final.0
       virtual packages : __osx=12.6=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/dd/miniforge3  (writable)
      conda av data dir : /Users/dd/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/dd/miniforge3/pkgs
                          /Users/dd/.conda/pkgs
       envs directories : /Users/dd/anaconda3/envs
                          /Users/dd/Library/r-miniconda
                          /Users/dd/Library/r-miniconda/envs
                          /Users/dd/miniforge3/envs
                          /Users/dd/.conda/envs
               platform : osx-arm64
             user-agent : conda/22.9.0 requests/2.28.1 CPython/3.9.13 Darwin/21.6.0 OSX/12.6
                UID:GID : 501:20
             netrc file : /Users/dd/.netrc
           offline mode : False

[hmaarrfk]

I can't spot anything immediately wrong with your environment.

It is quite possible that you are the first testing the package on conda forge.

It is quite possible that we have a bug in it.

[Conzel]

I have exactly the same problems on a M1 Mac. If it is helpful, I can also show some more information about my environment.

Meanwhile, I had a lot of problems building from scratch. I want to leave a step-by-step guide here so someone else who sees the problem has an easier time. I'm neither proficient with C-extensions nor CMake, so apologies if this is very hacky.

Building zfpy from scratch

First, clone ZFP repository. Then, do the following (for Python 3.10, change for other python versions accordingly):

mkdir build
cd build
cmake .. -DBUILD_ZFPY=ON \ 
-DCMAKE_BUILD_TYPE=RELEASE \
-DPYTHON_EXECUTABLE:FILEPATH="/opt/homebrew/bin/python3.10" \
-DPYTHON_LIBRARY="/opt/homebrew/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib" \
-DPYTHON_INCLUDE_DIR="/opt/homebrew/Frameworks/Python.framework/Versions/3.10/include/python3.10"      

Under bin you now find a .so library, which you can import with

import zfpy

if you started your python interpreter in the corresponding directory.

To make the package importable from anywhere, first go back to the project root. Then we change the setup.py file: find and remove the library_dirs attribute completely. Next, install the package with:

pip install .

Lastly, we manually copy the libs from the build process to a lib path on our system. This might look something like this:

cp build/lib/* /opt/homebrew/Caskroom/mambaforge/base/envs/climate/lib/

If you have no clue where to place the lib, just open up python and try to

import zfpy

Python will complain that it didn't find the library and will show you the content of your rpath.

[YijieYin]

Hi! I have the same issue as @dokato on my M1 mac. Thanks!

[jefferis]

See https://github.com/seung-lab/cloud-volume/issues/561 and https://github.com/seung-lab/cloud-volume/issues/570 for some workarounds @YijieYin

[hmaarrfk]

I've been skimming through https://github.com/LLNL/zfp/issues/175#issue-1338357172

I'm rebuilding the conda-forge package with the OSX_ARCHITECTURES flags specified https://github.com/conda-forge/zfpy-feedstock/pull/32

If somebody is still interested in conda-forge packages, they can follow the progress there. Other ideas are welcome too.