Installation issue on mac os

arogozhnikov commented 2 years ago

Hi, out-of-the box installation fails for me:

(pipeline) ➜  ~ pip install jxlpy
Collecting jxlpy
  Using cached jxlpy-0.9.1.tar.gz (73 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/axelr/system1/envs/pipeline/bin/python3 /Users/axelr/system1/envs/pipeline/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /var/folders/m7/d0p8pv5s6x5g09z3rwkgp24m0000gn/T/tmpt85aclzc
       cwd: /private/var/folders/m7/d0p8pv5s6x5g09z3rwkgp24m0000gn/T/pip-install-uxoxexbg/jxlpy_f0fdeceeee594351831466bb8cfd84aa
  Complete output (22 lines):
  Traceback (most recent call last):
    File "/Users/axelr/system1/envs/pipeline/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/Users/axelr/system1/envs/pipeline/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/axelr/system1/envs/pipeline/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/m7/d0p8pv5s6x5g09z3rwkgp24m0000gn/T/pip-build-env-36rlrw9c/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/private/var/folders/m7/d0p8pv5s6x5g09z3rwkgp24m0000gn/T/pip-build-env-36rlrw9c/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/m7/d0p8pv5s6x5g09z3rwkgp24m0000gn/T/pip-build-env-36rlrw9c/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 37, in <module>
      ext_modules=cythonize([jxlpy_ext]),
    File "/private/var/folders/m7/d0p8pv5s6x5g09z3rwkgp24m0000gn/T/pip-build-env-36rlrw9c/overlay/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 966, in cythonize
      module_list, module_metadata = create_extension_list(
    File "/private/var/folders/m7/d0p8pv5s6x5g09z3rwkgp24m0000gn/T/pip-build-env-36rlrw9c/overlay/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 816, in create_extension_list
      for file in nonempty(sorted(extended_iglob(filepattern)), "'%s' doesn't match any files" % filepattern):
    File "/private/var/folders/m7/d0p8pv5s6x5g09z3rwkgp24m0000gn/T/pip-build-env-36rlrw9c/overlay/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 114, in nonempty
      raise ValueError(error_msg)
  ValueError: '_jxlpy/_jxl.pyx' doesn't match any files

Environment:

Python 3.9.0
pip 21.3.1
mac os 11.5.2

EraYaN commented 2 years ago

This also does not work for windows, and to make this really work prebuilt wheels would have to be provided.

olokelo commented 2 years ago

Hi!

Sadly, I don't have any Mac OS system and I'm unfamiliar with development there. Provided builds on PyPI for now are available only for Linux, if you have a different OS then pip will try to build the package from source.

I'm not sure how to build libjxl on Mac OS, but you could try to do it. It would be the first step to getting jxlpy working.

@EraYaN is right, I would have to configure making prebuilt wheels for Mac OS and Windows in Github actions and publishing them to PyPI.

EraYaN commented 2 years ago

On Windows I just let vcpkg deal with it, and that seems to run on macOS as well. And there instead of auditwheel or delvewheel you should be able to use delocate to fix the wheel after building. For windows this mostly works apart from #7

cmahnke commented 5 months ago

I had issues as well, but installing from source works for me:

First install JXL using brew.
```
brew install jpeg-xl
```
Make sure you're using Python from brew as well, currently Python 3.11 is the default. Then build from source using pip, make sure to include the right compiler settings (environment variables LDFLAGS and CPPFLAGS):
```
LDFLAGS="-L/opt/homebrew/lib/" CPPFLAGS="-I/opt/homebrew/include/" /opt/homebrew/bin/pip3.11 install git+https://github.com/olokelo/jxlpy
```


Optional:
* Install Pillow using `pip` to use JXL from PIL

/opt/homebrew/bin/pip3.11 install Pillow

olokelo commented 5 months ago

@cmahnke Thank you for testing that.

cmahnke commented 5 months ago

@olokelo From my point of view the interesting question is, why it fails when using pip and the name of the package. The error is still the initial one:

Using pip 24.0 from /opt/homebrew/lib/python3.11/site-packages/pip (python 3.11)
WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/numpy-1.26.4.dist-info due to invalid metadata entry 'name'
WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/numpy-1.26.4.dist-info due to invalid metadata entry 'name'
WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/argcomplete-3.2.2.dist-info due to invalid metadata entry 'name'
WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/MarkupSafe-2.1.4.dist-info due to invalid metadata entry 'name'
WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/MarkupSafe-2.1.5.dist-info due to invalid metadata entry 'name'
Collecting jxlpy
  Using cached jxlpy-0.9.3.tar.gz (101 kB)
  Running command pip subprocess to install build dependencies
  Using pip 24.0 from /opt/homebrew/lib/python3.11/site-packages/pip (python 3.11)
  WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/numpy-1.26.4.dist-info due to invalid metadata entry 'name'
  Collecting setuptools
    Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/55/3a/5121b58b578a598b269537e09a316ad2a94fdd561a2c6eb75cd68578cc6b/setuptools-69.0.3-py3-none-any.whl.metadata
    Using cached setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB)
  Collecting wheel
    Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/c7/c3/55076fc728723ef927521abaa1955213d094933dc36d4a2008d5101e1af5/wheel-0.42.0-py3-none-any.whl.metadata
    Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB)
  Collecting Cython
    Obtaining dependency information for Cython from https://files.pythonhosted.org/packages/e3/7f/f584f5d15323feb897d42ef0e9d910649e2150d7a30cf7e7a8cc1d236e6f/Cython-3.0.8-py2.py3-none-any.whl.metadata
    Using cached Cython-3.0.8-py2.py3-none-any.whl.metadata (3.2 kB)
  Using cached setuptools-69.0.3-py3-none-any.whl (819 kB)
  Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
  Using cached Cython-3.0.8-py2.py3-none-any.whl (1.2 MB)
  WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/numpy-1.26.4.dist-info due to invalid metadata entry 'name'
  WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/argcomplete-3.2.2.dist-info due to invalid metadata entry 'name'
  WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/MarkupSafe-2.1.4.dist-info due to invalid metadata entry 'name'
  WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/MarkupSafe-2.1.5.dist-info due to invalid metadata entry 'name'
  Installing collected packages: wheel, setuptools, Cython
    Creating /private/var/folders/k1/hdlmqrhx72vf59cyh6tkys9h0000gn/T/pip-build-env-u1qeng45/overlay/bin
    changing mode of /private/var/folders/k1/hdlmqrhx72vf59cyh6tkys9h0000gn/T/pip-build-env-u1qeng45/overlay/bin/wheel to 755
    changing mode of /private/var/folders/k1/hdlmqrhx72vf59cyh6tkys9h0000gn/T/pip-build-env-u1qeng45/overlay/bin/cygdb to 755
    changing mode of /private/var/folders/k1/hdlmqrhx72vf59cyh6tkys9h0000gn/T/pip-build-env-u1qeng45/overlay/bin/cython to 755
    changing mode of /private/var/folders/k1/hdlmqrhx72vf59cyh6tkys9h0000gn/T/pip-build-env-u1qeng45/overlay/bin/cythonize to 755
  Successfully installed Cython-3.0.8 setuptools-69.0.3 wheel-0.42.0
  WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/numpy-1.26.4.dist-info due to invalid metadata entry 'name'
  WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/numpy-1.26.4.dist-info due to invalid metadata entry 'name'
  WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/numpy-1.26.4.dist-info due to invalid metadata entry 'name'
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Traceback (most recent call last):
    File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 37, in <module>
    File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1010, in cythonize
      module_list, module_metadata = create_extension_list(
                                     ^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 845, in create_extension_list
      for file in nonempty(sorted(extended_iglob(filepattern)), "'%s' doesn't match any files" % filepattern):
    File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 117, in nonempty
      raise ValueError(error_msg)
  ValueError: '_jxlpy/_jxl.pyx' doesn't match any files
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/homebrew/opt/python@3.11/bin/python3.11 /opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /var/folders/k1/hdlmqrhx72vf59cyh6tkys9h0000gn/T/tmpd3i8twho
  cwd: /private/var/folders/k1/hdlmqrhx72vf59cyh6tkys9h0000gn/T/pip-install-76c90nj_/jxlpy_84196f1b60b449a6972c8f5e73a20908
  Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

From the back of your head, can you say if the file _jxlpy/_jxl.pyx is missing from the disto?

olokelo commented 5 months ago

@cmahnke Thanks for pointing that out. Indeed it seems like only the cythonized _jxlpy/_jxl.cpp ends up in the source distribution.

I've checked that running sdist command of setup.py:

$ python setup.py sdist
Compiling _jxlpy/_jxl.pyx because it changed.
[1/1] Cythonizing _jxlpy/_jxl.pyx
running sdist
running egg_info
writing jxlpy.egg-info/PKG-INFO
writing dependency_links to jxlpy.egg-info/dependency_links.txt
writing requirements to jxlpy.egg-info/requires.txt
writing top-level names to jxlpy.egg-info/top_level.txt
reading manifest file 'jxlpy.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'jxlpy.egg-info/SOURCES.txt'
running check
creating jxlpy-0.9.3
creating jxlpy-0.9.3/_jxlpy
creating jxlpy-0.9.3/jxlpy
creating jxlpy-0.9.3/jxlpy.egg-info
copying files to jxlpy-0.9.3...
copying LICENSE -> jxlpy-0.9.3
copying README.md -> jxlpy-0.9.3
copying pyproject.toml -> jxlpy-0.9.3
copying setup.py -> jxlpy-0.9.3
copying _jxlpy/_jxl.cpp -> jxlpy-0.9.3/_jxlpy
copying jxlpy/JXLImagePlugin.py -> jxlpy-0.9.3/jxlpy
copying jxlpy/__init__.py -> jxlpy-0.9.3/jxlpy
copying jxlpy.egg-info/PKG-INFO -> jxlpy-0.9.3/jxlpy.egg-info
copying jxlpy.egg-info/SOURCES.txt -> jxlpy-0.9.3/jxlpy.egg-info
copying jxlpy.egg-info/dependency_links.txt -> jxlpy-0.9.3/jxlpy.egg-info
copying jxlpy.egg-info/requires.txt -> jxlpy-0.9.3/jxlpy.egg-info
copying jxlpy.egg-info/top_level.txt -> jxlpy-0.9.3/jxlpy.egg-info
copying jxlpy.egg-info/SOURCES.txt -> jxlpy-0.9.3/jxlpy.egg-info
Writing jxlpy-0.9.3/setup.cfg
Creating tar archive
removing 'jxlpy-0.9.3' (and everything under it)

I have no idea why that's the case but I made a dirty hack and it seems installing from source and binary distribution can work if I change the sources in setup.py to (importing the os module before it of course):

    sources=["_jxlpy/_jxl.pyx" if os.path.exists("_jxlpy/_jxl.pyx") else "_jxlpy/_jxl.cpp"],

Of course there has to be a cleaner way of achieving that.

cmahnke commented 5 months ago

I suspect that this is only testable if the change has been pushed to PIPy, but I wouldn't do a release just for this... Maybe It's possible to get some PRs from the forks mentioned here before creating a new release.

arogozhnikov commented 5 months ago

not only, but there is test pypi: https://test.pypi.org/

olokelo commented 5 months ago

@arogozhnikov thank you, I didn't know about test pypi. @cmahnke I might do a release for that, it's not a huge problem especially that libjxl 0.9.2 got released just after I published jxlpy 0.9.3 which still used libjxl 0.9.1 (sorry about this version confusion between jxlpy and libjxl right now that's unlucky) so it wouldn't hurt if I included new libjxl in wheels on pypi. Of course I'm also open to any pull requests that would help with development or packaging.

olokelo commented 4 months ago

@cmahnke I've made a release 0.9.4 which seems to fix this error.

cmahnke commented 4 months ago

@olokelo: Great! It works now, using libjxl (version 0.10.1) from brew works now. Note: Pointing to the right includes is still required:

LDFLAGS="-L/opt/homebrew/lib/" CPPFLAGS="-I/opt/homebrew/include/" pip3 install jxlpy

But I would consider this a minor inconvenience, since getting reliably around this is a bit of work, without a huge benefit. So, this needs to be documented.

Since @arogozhnikov is the original reporter we should wait (maybe for a week) for his feedback and close otherwise.

arogozhnikov commented 4 months ago

I can confirm that this instruction: https://github.com/olokelo/jxlpy/issues/6#issuecomment-1932603158 with new libjxl did install successfully.

Also checked:

encode_simple - ok
decode_simple - ok
pillow tests - fail, probably should register jxlpy somehow, did not get to it

olokelo / jxlpy

Installation issue on mac os #6