Cramjam wrong architecture error when installing on M1 with conda (mambaforge)

[avriiil]

[X] I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

I'm working on Apple M1 with a mambaforge deployment of conda. When trying to save a dataframe to parquet using fastparquet I'm getting the following error:

ImportError: dlopen(/Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so, 2): no suitable image found. Did find: /Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so: mach-o, but wrong architecture /Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so: mach-o, but wrong architecture Seems like conda is giving me the wrong version of cramjam?

MRE:

conda create -n cramjam conda install cramjam ipython ipython import cramjam

ImportError: dlopen(/Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so, 2): no suitable image found. Did find: /Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so: mach-o, but wrong architecture /Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so: mach-o, but wrong architecture

import cramjam does work when doing a pip install cramjam. But fails again if I then do a subsequent conda install fastparquet.

$ conda list
# Name                    Version                   Build  Channel
ca-certificates           2021.5.30            h4653dfc_0    conda-forge
cramjam                   2.3.2                    pypi_0    pypi
fastparquet               0.7.1            py39h20ef057_0    conda-forge
fsspec                    2021.8.1           pyhd8ed1ab_0    conda-forge
libblas                   3.9.0           11_osxarm64_openblas    conda-forge
libcblas                  3.9.0           11_osxarm64_openblas    conda-forge
libcxx                    12.0.1               h168391b_0    conda-forge
libgfortran               5.0.0.dev0      11_0_1_hf114ba7_23    conda-forge
libgfortran5              11.0.1.dev0         hf114ba7_23    conda-forge
liblapack                 3.9.0           11_osxarm64_openblas    conda-forge
libopenblas               0.3.17          openmp_h5dd58f0_1    conda-forge
llvm-openmp               12.0.1               hf3c4609_1    conda-forge
ncurses                   6.2                  h9aa5885_4    conda-forge
numpy                     1.21.2           py39h1f3b974_0    conda-forge
openssl                   1.1.1l               h3422bc3_0    conda-forge
pandas                    1.3.2            py39h7f752ed_0    conda-forge
pip                       21.2.4             pyhd8ed1ab_0    conda-forge
python                    3.9.7           h54d631c_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
readline                  8.1                  hedafd6a_0    conda-forge
setuptools                57.4.0           py39h2804cbe_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.36.0               h72a2b83_0    conda-forge
thrift                    0.13.0           py39h93dc1e2_2    conda-forge
tk                        8.6.11               he1e0b03_1    conda-forge
tzdata                    2021a                he74cb21_1    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
zlib                      1.2.11            h31e879b_1009    conda-forge

$ conda info

    active environment : cramjam
    active env location : /Users/rpelgrim/mambaforge/envs/cramjam
            shell level : 2
       user config file : /Users/rpelgrim/.condarc
 populated config files : /Users/rpelgrim/mambaforge/.condarc
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.9.6.final.0
       virtual packages : __osx=11.3.1=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/rpelgrim/mambaforge  (writable)
      conda av data dir : /Users/rpelgrim/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/rpelgrim/mambaforge/pkgs
                          /Users/rpelgrim/.conda/pkgs
       envs directories : /Users/rpelgrim/mambaforge/envs
                          /Users/rpelgrim/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.10.3 requests/2.26.0 CPython/3.9.6 Darwin/20.4.0 OSX/11.3.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[avriiil]

see discussion with @milesgranger in this issue for more context

[avriiil]

Is there a way to have the conda install fastparquet use the existing version of cramjam that I got from pip?

[milesgranger]

Unless @martindurant has a solution, I can poke around at this maybe sometime this week or next.

Martin is probably annoyed with me by now about this :sweat_smile: , I'll keep saying that it seems weird conda/recepies can't play well with an already pre-compiled binary. Especially b/c it works in this case (installing it separately from PyPI) and these specific wheels have zero Python dependencies. :disappointed: We're already at a point that we have to retry builds b/c they may end up on less powerful VMs and exceeds an hour limit; vs simply taking the already existing wheels from PyPI.

I really like having it distributed on conda, but it's a pain to do so in this case. "There must be a better way."

[martindurant]

I am asking some conda people, we'll see. Indeed, for fastparquet I have always found the conda path smoother than the wheel one (we didn't have non-source distribution for a long time).

[katietz]

This issue is mainly caused by the python interpreter and is skipping of .py->.pyc on osx-arm64. This happens due issue of cross-compiling and building python. I tested described issue on defaults after building maturin (downgraded to use rust >= 1.53.0), and cranjam. The error seems not to exists there for me. We don't provide those two packages on defaults for now, so I would recomment for the time being to use either pip (might not work for fastparquet). I provide the test versions in my private channel 'ktietz' if someone is interested in them.

[avriiil]

@katietz - removing fastparquet from my environment.yml and installing it through pip did the trick here for me. Thanks so much!

[milesgranger]

Nice! I also wonder if something like this would work for you?

name: test
channels:
  - conda-forge
dependencies:
  - fastparquet
  - pip:
    - cramjam

[avriiil]

@milesgranger - yes, that works too. thanks! 🙏

[erykoff]

@rrpelgrim This should now be fixed with the latest builds of cramjam-2.5.0. Please confirm on your end, and if so this issue can be closed. Thanks!

[martindurant]

Should we remove the previous conda release that installs incorrectly? That can be done by submitting a patch PR, if we think it's important (I've never done this before myself)

[erykoff]

I think we need to mark the previous osx-arm64 builds as "broken", actually. I'll look into that.

[martindurant]

feel free to ping conda-forge admins, if you need help.

[erykoff]

https://github.com/conda-forge/admin-requests/pull/359