search

libvips / pyvips

python binding for libvips using cffi
MIT License
649 stars 50 forks source link

Add optional dependency on `pyvips-binary` #507

Closed kleisauke closed 3 weeks ago

kleisauke commented 2 months ago

Allowing users to install pyvips with binary packages via:

$ pip install "pyvips[binary]"

See: https://pypi.org/project/pyvips-binary/.

Marked as draft due to this notes:

jcupitt commented 2 months ago

Nice!

Yes, maybe a separate repo is better.

kleisauke commented 2 months ago

Done!

https://pypi.org/project/pyvips-binary/ https://github.com/kleisauke/pyvips-binary

kleisauke commented 2 months ago

... here are some testing notes:

$ pkg-config --exists --print-errors vips
Package vips was not found in the pkg-config search path.
Perhaps you should add the directory containing `vips.pc'
to the PKG_CONFIG_PATH environment variable
Package 'vips', required by 'virtual:world', not found
$ pip3 install pyvips
$ python -c "import pyvips; print(pyvips.API_mode)"
ModuleNotFoundError: No module named '_libvips'

During handling of the above exception, another exception occurred:

[...]

OSError: cannot load library 'libvips.so.42': libvips.so.42: cannot open shared object file: No such file or directory.

[...]
$ pip3 install pyvips-binary
$ python -c "import pyvips; print(pyvips.API_mode)"
True
$ python -c "import pyvips; pyvips.Image.black(1024, 1024).write_to_file('x.jpg')"
$ file x.jpg
x.jpg: JPEG image data, Exif standard: [TIFF image data, little-endian, direntries=6, orientation=upper-left, xresolution=86, yresolution=94, resolutionunit=2], baseline, precision 8, 1024x1024, components 1
jcupitt commented 2 months ago

Ah fantastic! That's so cool. I'll test here as well.

Shall we put you on the authors list as well? What email addr would you prefer?

jcupitt commented 2 months ago

Is is possible to invert the sense of this option? I'm sure most people should use the netvips binaries rather than the system ones.

So pip install pyvips has the pyvips-binary dependency, and pip install pyvips[system] (something like that?) will use the system libvips.

kleisauke commented 2 months ago

Shall we put you on the authors list as well? What email addr would you prefer?

Feel free to add me to the authors list! I usually use the same email address that's on my GitHub profile. https://github.com/kleisauke/pyvips-binary/blob/deaf1bf1c360e400f8b7b7869d05c4e695c2b4e5/pyproject.toml#L17

Is is possible to invert the sense of this option? I'm sure most people should use the netvips binaries rather than the system ones.

I had the same thought, but it doesn't seem to be possible at the moment. This future possible enhancement is discussed in detail here: https://discuss.python.org/t/4898.

jcupitt commented 2 months ago

Maybe we could just make another pyvips package (pyvips-system?) without a dependency on pyvips-binary?

kleisauke commented 2 months ago

That approach might work, but I'm uncertain if it's the best option for users who prefer relying on system binaries.

For example, Zulip tends to favor using libvips from the package manager (see e.g. commit https://github.com/zulip/zulip/commit/e8f7e281a7d389297783a2185fd0ff09ea4493f5). Additionally, pyvips is commonly used by users who process whole-slide images, something that the pyvips-binary dependency doesn't support (since it uses the -web variant).

The pyvips-binary package also lacks pre-built wheels for less common architectures (such as linux-riscv64, linux-ppc64le, win32-arm64, etc.) and only supports CPython.

Perhaps we should consider catching the OSError mentioned above and re-throwing it with a message suggesting that the optional [binary] package might need to be installed? sharp handles this elegantly, see for example:

Details ``` node_modules\sharp\lib\sharp.js:114 throw new Error(help.join('\n')); ^ Error: Could not load the "sharp" module using the win32-arm64 runtime Possible solutions: - Manually install libvips >= 8.15.3 - Add experimental WebAssembly-based dependencies: npm install --cpu=wasm32 sharp npm install @img/sharp-wasm32 - Consult the installation documentation: See https://sharp.pixelplumbing.com/install at Object. (node_modules\sharp\lib\sharp.js:114:9) at Module._compile (node:internal/modules/cjs/loader:1460:14) at Module._extensions..js (node:internal/modules/cjs/loader:1544:10) at Module.load (node:internal/modules/cjs/loader:1275:32) at Module._load (node:internal/modules/cjs/loader:1091:12) at wrapModuleLoad (node:internal/modules/cjs/loader:212:19) at Module.require (node:internal/modules/cjs/loader:1297:12) at require (node:internal/modules/helpers:123:16) at Object. (node_modules\sharp\lib\constructor.js:10:1) at Module._compile (node:internal/modules/cjs/loader:1460:14) ```
kleisauke commented 1 month ago

This is ready for review now. The last todo note has been fixed with https://pypi.org/project/pyvips-binary/8.16.0rc1/.

kleisauke commented 3 weeks ago

https://pypi.org/project/pyvips-binary/8.16.0/ :tada:

jcupitt commented 3 weeks ago

Hooray! This is really great Kleis.

jcupitt commented 3 weeks ago

Shall we do 3.0.0? I think everything is ready now.

kleisauke commented 3 weeks ago

Sounds good to me, feel free to release 3.0.0 (coincidentally NetVips 3.0.0 was also released today :tada:).

jcupitt commented 3 weeks ago

Congratulations!

... I'm being dumb, I don't seem to be able to get pyvips to find its libvips on ubuntu:

$ pip install pyvips-binary==8.16.0
...
Downloading pyvips_binary-8.16.0-cp37-abi3-manylinux_2_28_x86_64.whl (7.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 9.0 MB/s eta 0:00:00
Installing collected packages: pyvips-binary
Successfully installed pyvips-binary-8.16.0
$

But it makes a binary with an extra (I guess?) commit number:

$ find ~/testing -name "*libvips*"
./python3.12/site-packages/pyvips_binary.libs/libvips-541602e6.so.42

Then:

$ python3
Python 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
Traceback (most recent call last):
  File "/home/john/GIT/pyvips/pyvips/__init__.py", line 30, in <module>
    import _libvips
ImportError: libvips.so.42: cannot open shared object file: No such file or directory
kleisauke commented 3 weeks ago

I wonder if this relates to https://github.com/libvips/libvips/issues/3901#issuecomment-2009154519, which can also be fixed by running pip cache purge. What does ldd $(python -m site --user-site)/_libvips.abi3.so print? I see:

$ pip install pyvips
Defaulting to user installation because normal site-packages is not writeable
Collecting pyvips
  Using cached pyvips-2.2.3-py2.py3-none-any.whl
Requirement already satisfied: cffi>=1.0.0 in /usr/lib64/python3.12/site-packages (from pyvips) (1.16.0)
Requirement already satisfied: pycparser in ./.local/lib/python3.12/site-packages (from cffi>=1.0.0->pyvips) (2.22)
Installing collected packages: pyvips
Successfully installed pyvips-2.2.3
$ pip install pyvips-binary
Defaulting to user installation because normal site-packages is not writeable
Collecting pyvips-binary
  Downloading pyvips_binary-8.16.0-cp37-abi3-manylinux_2_28_x86_64.whl.metadata (2.3 kB)
Requirement already satisfied: cffi>=1.0.0 in /usr/lib64/python3.12/site-packages (from pyvips-binary) (1.16.0)
Requirement already satisfied: pycparser in ./.local/lib/python3.12/site-packages (from cffi>=1.0.0->pyvips-binary) (2.22)
Downloading pyvips_binary-8.16.0-cp37-abi3-manylinux_2_28_x86_64.whl (7.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 8.9 MB/s eta 0:00:00
Installing collected packages: pyvips-binary
Successfully installed pyvips-binary-8.16.0
$ ldd $(python -m site --user-site)/_libvips.abi3.so
    linux-vdso.so.1 (0x00007fd3d253a000)
    libvips-541602e6.so.42 => /home/kleisauke/.local/lib/python3.12/site-packages/pyvips_binary.libs/libvips-541602e6.so.42 (0x00007fd3d1200000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd3d24f4000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fd3d100f000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd3d24e2000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fd3d24dd000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fd3d0c00000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fd3d23f7000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd3d23c9000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd3d253c000)
$ python3
Python 3.12.7 (main, Oct  1 2024, 00:00:00) [GCC 14.2.1 20240912 (Red Hat 14.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
>>> print(pyvips.API_mode)
True
>>>
kleisauke commented 3 weeks ago

The naming of libvips-<unique>.so.42 happens automagically as part of the auditwheel process. Comment https://github.com/lovell/sharp/issues/4095#issuecomment-2394260539 has some background details on this.