Rebuild for Apple Silicon?

[axu2]

When I try to import mozjpeg_lossless_optimization

I get

ImportError: dlopen(/Users/name/Documents/GitHub/kcc/venv/lib/python3.9/site-packages/mozjpeg_lossless_optimization/_mozjpeg_opti.abi3.so, 0x0002): 
symbol not found in flat namespace '_jpeg_copy_critical_parameters'

Can anyone else replicate this? I feel like it probably just needs to be rebuilt and put on pip. I suspect that it is an Apple Silicon issue.

[flozz]

Hello,

Sadly I have no macOS device to debug this...

Maybe removing the universal2 target for macOS could fix the issue (keeping only x86_64 and arm64)?

https://github.com/wanadev/mozjpeg-lossless-optimization/blob/master/.github/workflows/python-packages.yml#LL60C42-L60C52

[flozz]

Can you try to install the arm64 wheel corresponding to your Python version to check if it works better than the universal2 one?

https://pypi.org/project/mozjpeg-lossless-optimization/#files

[axu2]

I'm definitely using the arm64 version:

(venv) alex@mac-mini kcc % pip install mozjpeg_lossless_optimization
Collecting mozjpeg_lossless_optimization
  Using cached mozjpeg_lossless_optimization-1.1.2-cp311-cp311-macosx_11_0_arm64.whl (10.0 kB)
Requirement already satisfied: cffi>=1.0.0 in ./venv/lib/python3.11/site-packages (from mozjpeg_lossless_optimization) (1.15.1)
Requirement already satisfied: pycparser in ./venv/lib/python3.11/site-packages (from cffi>=1.0.0->mozjpeg_lossless_optimization) (2.21)
Installing collected packages: mozjpeg_lossless_optimization
Successfully installed mozjpeg_lossless_optimization-1.1.2

(venv) alex@mac-mini kcc % python
Python 3.11.3 (v3.11.3:f3909b8bc8, Apr  4 2023, 20:12:10) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mozjpeg_lossless_optimization
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/alex/Documents/GitHub/kcc/venv/lib/python3.11/site-packages/mozjpeg_lossless_optimization/__init__.py", line 1, in <module>
    from .mozjpeg_opti import optimize
  File "/Users/alex/Documents/GitHub/kcc/venv/lib/python3.11/site-packages/mozjpeg_lossless_optimization/mozjpeg_opti.py", line 1, in <module>
    from ._mozjpeg_opti import lib, ffi
ImportError: dlopen(/Users/alex/Documents/GitHub/kcc/venv/lib/python3.11/site-packages/mozjpeg_lossless_optimization/_mozjpeg_opti.abi3.so, 0x0002): symbol not found in flat namespace '_jpeg_CreateCompress'

(venv) alex@mac-mini kcc % pip freeze
cffi==1.15.1
install==1.3.5
mozjpeg-lossless-optimization==1.1.2
pycparser==2.21

[flozz]

Ok :(

We will have more work then...

Can you test installing from the sdist package (so you will have to compile the lib on your own machine)? That way we will be able to check if it is a build issue on Github Actions or a problem with the lib itself (I never tested it on macOS ^^')

I know I had some issue on an other project because the build process not cleaned the build directory between architectures:

https://github.com/wanadev/pyguetzli/blob/master/.github/workflows/python-packages.yml#LL62C64-L62C64

[axu2]

Mind telling me the exact commands I need? I've only ever installed stuff from pip or with a requirements.txt

[flozz]

Yes of course! :)

You can install it with the following command:

pip install https://files.pythonhosted.org/packages/62/cb/d73282f2e85058ceeb1dd19f4251ec7d47c05cff4b14d6b51e34a4f96c9e/mozjpeg-lossless-optimization-1.1.2.tar.gz

You will need build tools (cmake, and probably XCode on macOS?) and then everything should be automated :)

[axu2]

Works fine after installing cmake.

alex@mac-mini ~ % brew install cmake
Running `brew update --auto-update`...
==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
  https://github.com/Homebrew/brew#donations

Warning: Treating cmake as a formula. For the cask, use homebrew/cask/cmake
==> Fetching cmake
==> Downloading https://ghcr.io/v2/homebrew/core/cmake/manifests/3.26.4
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/cmake/blobs/sha256:68ab4f8a848b
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sh
######################################################################### 100.0%
==> Pouring cmake--3.26.4.arm64_ventura.bottle.tar.gz
==> Caveats
To install the CMake documentation, run:
  brew install cmake-docs

Emacs Lisp files have been installed to:
  /opt/homebrew/share/emacs/site-lisp/cmake
==> Summary
🍺  /opt/homebrew/Cellar/cmake/3.26.4: 3,188 files, 45.5MB
==> Running `brew cleanup cmake`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

(venv) alex@mac-mini kcc % pip install https://files.pythonhosted.org/packages/62/cb/d73282f2e85058ceeb1dd19f4251ec7d47c05cff4b14d6b51e34a4f96c9e/mozjpeg-lossless-optimization-1.1.2.tar.gz
Collecting https://files.pythonhosted.org/packages/62/cb/d73282f2e85058ceeb1dd19f4251ec7d47c05cff4b14d6b51e34a4f96c9e/mozjpeg-lossless-optimization-1.1.2.tar.gz
  Using cached mozjpeg-lossless-optimization-1.1.2.tar.gz (1.0 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: cffi>=1.0.0 in ./venv/lib/python3.11/site-packages (from mozjpeg-lossless-optimization==1.1.2) (1.15.1)
Requirement already satisfied: pycparser in ./venv/lib/python3.11/site-packages (from cffi>=1.0.0->mozjpeg-lossless-optimization==1.1.2) (2.21)
Building wheels for collected packages: mozjpeg-lossless-optimization
  Building wheel for mozjpeg-lossless-optimization (pyproject.toml) ... done
  Created wheel for mozjpeg-lossless-optimization: filename=mozjpeg_lossless_optimization-1.1.2-cp311-cp311-macosx_10_9_universal2.whl size=91411 sha256=b113af7ae2388489159ba61f0c7b1176ead8302d623e6cad02e47dabf6b3686a
  Stored in directory: /Users/alex/Library/Caches/pip/wheels/82/2f/4f/55e19af8e91176d5244cae573c1168febe308f06be0b22dfbb
Successfully built mozjpeg-lossless-optimization
Installing collected packages: mozjpeg-lossless-optimization
Successfully installed mozjpeg-lossless-optimization-1.1.2

(venv) alex@mac-mini kcc % python
Python 3.11.3 (v3.11.3:f3909b8bc8, Apr  4 2023, 20:12:10) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mozjpeg_lossless_optimization
>>> 

[flozz]

Ok so the library can be built on macOS and work with this OS, so the problem come from the GH Actions build.

---

I created a branch (5-macos-wheel) to be able to tweak the build and see if it work better.

Here are the wheels I just built (workflow):

• wheels(1).zip

Can you try to install the *-macosx_11_0_arm64.whl file corresponding to your Python version?

pip install mozjpeg_lossless_optimization-1.1.2-cp311-cp311-macosx_11_0_arm64.whl

Does it work?

---

If the wheel from Github Actions does not work, can you build a wheel on your machine, test it, and share it with me so I can try to see if I spot some differences?

To build the wheel:

git clone https://github.com/wanadev/mozjpeg-lossless-optimization.git
cd mozjpeg-lossless-optimization/
python3 -m venv __env__
source __env__/bin/activate
pip install wheel
python setup.py bdist_wheel

The resulting package goes in dist/ folder

[axu2]

Needed to build it. This one works fine.

mozjpeg_lossless_optimization-1.1.2-cp311-cp311-macosx_10_9_universal2.whl.zip

This one can be pip installed fine.

[axu2]

@flozz Do you think changing macos-11 to macos-latest would help in the build file? Maybe a bug was fixed between 11 and 13.

Also, are you using a self hosted runner? Maybe when GitHub adds m1 runners later this year it will work then.

Otherwise just installing from source is fine like you showed me.

[flozz]

Hello,

I have no time for now (and I have no macOS 11 PC to check if it works) but it is a thing we can test...

We use the GitHub runners (and it should already be able to build for M1 but as we saw, it seems it does not work very well...)

[axu2]

Actually, I uninstalled everything in homebrew, I had some jpeg related stuff installed. I also uninstalled homebrew python, using python.org version instead. Now it works. No clue why.

[flozz]

Ok, Good to know. :)

I hope not too many people will have this issue... I have no Mac to work on it anyway ^^'

[axu2]

Based on the discussion in the other issue, seems like it was a python version issue. The latest 3.11.4 works, homebrew or not.