search

libvips / pyvips

python binding for libvips using cffi
MIT License
649 stars 50 forks source link

new release when & ship with libvips? #443

Open SteveHawk opened 11 months ago

SteveHawk commented 11 months ago

Hi, I'm just wondering, is there going to be a new release of pyvips soon?

libvips 8.15 introduces highway for simd which is exciting, however current pyvips v2.2.1 on PyPI only supports libvips up to 8.13. Master branch of pyvips does work with 8.15 in my setup, but it would be really nice to have a stable version to lock in.

Oh and, will it be possible for pyvips to ship with prebuilt libvips in the future? As my understanding, net-vips and sharp has been doing this for years, I tried the prebuilt package from kleisauke/libvips-packaging with pyvips and it works like a charm. Using pyvips without needing to install many dozens of system dependencies is a lifesaver in deployment.

Thanks!

jcupitt commented 11 months ago

Hi @SteveHawk,

pyvips works with any libvips version -- it introspects the libvips binary and exposes the API it finds. pyvips v2.2.1 should work fine with 8.15 and 8.16.

Yes, we've talked about including a binary, and you're right it'd be pretty simple. Someone just needs to do the work :( pyvips still uses the prehistoric setup.py, so really the whole installer needs redoing and updating.

One slight issue is that pyvips supports ABI (portable but slow and leaky) and API (fast and stable) modes, but with a prebuilt libvips binary we'd be stuck in ABI mode. Maybe the benefits are worth it.

Using pyvips without needing to install many dozens of system dependencies is a lifesaver in deployment.

You should just need the libvips-dev package (or equivalent), I think. python-dev as well I guess, and a C compiler.

SteveHawk commented 11 months ago

pyvips works with any libvips version

Oh that's weird. I asked because I tried to use pyvips v2.2.1 with 8.15, but didn't work and throw errors immediately (seems to be related to glib). The master branch works though, that's why I thought v2.2.1 is not compatible with 8.15.

Here's how I did it:

After setting up the environment, enter python shell and run:

>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> from pyvips import Image
DEBUG:pyvips:Binary module load failed: /usr/local/lib/python3.11/site-packages/_libvips.abi3.so: undefined symbol: vips_path_mode7
DEBUG:pyvips:Falling back to ABI mode
DEBUG:pyvips:Loaded lib <cffi.api._make_ffi_library.<locals>.FFILibrary object at 0x7fa8935a3550>
DEBUG:pyvips:Loaded lib <cffi.api._make_ffi_library.<locals>.FFILibrary object at 0x7fa8936f3710>
DEBUG:pyvips:Inited libvips
>>> Image.new_from_file("/root/lib/test.png")
DEBUG:pyvips.voperation:VipsOperation.call: operation_name = VipsForeignLoadPngFile
DEBUG:pyvips.voperation:VipsOperation.call: match_image = None
DEBUG:pyvips.vobject:VipsObject.set: name = filename, value = /root/lib/test.png

(process:875): GLib-CRITICAL **: 09:49:38.370: g_datalist_id_set_data_full: assertion 'key_id > 0' failed

(process:875): GLib-GObject-CRITICAL **: 09:49:38.370: g_param_spec_pool_lookup: assertion 'pool != NULL' failed

(process:875): GLib-GObject-WARNING **: 09:49:38.370: g_object_set_is_valid_property: object class '(null)' has no property named 'filename'
Segmentation fault (core dumped)

That crashed. I also tried thumbnail_buffer, with no luck:

>>> Image.thumbnail_buffer(b"test", 128)
DEBUG:pyvips.voperation:VipsOperation.call: operation_name = thumbnail_buffer
DEBUG:pyvips.voperation:VipsOperation.call: match_image = None
DEBUG:pyvips.vobject:VipsObject.set: name = buffer, value = b'test'
DEBUG:pyvips.error:Error unsupported gtype for set NULL, fundamental GBoxed 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/pyvips/vimage.py", line 256, in call_function
    return pyvips.Operation.call(name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pyvips/voperation.py", line 282, in call
    op.set(name, intro.details[name]['flags'], match_image, value)
  File "/usr/local/lib/python3.11/site-packages/pyvips/voperation.py", line 216, in set
    super(Operation, self).set(name, value)
  File "/usr/local/lib/python3.11/site-packages/pyvips/vobject.py", line 122, in set
    gv.set(value)
  File "/usr/local/lib/python3.11/site-packages/pyvips/gvalue.py", line 242, in set
    raise Error('unsupported gtype for set {0}, fundamental {1}'.
pyvips.error.Error: unsupported gtype for set NULL, fundamental GBoxed

When I uninstall the pypi version and install the master branch, pip show still reports API mode, and everything works as expected:

>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> from pyvips import Image
DEBUG:pyvips:Loaded binary module _libvips
DEBUG:pyvips:Module generated for libvips 8.15
DEBUG:pyvips:Linked to libvips 8.15
DEBUG:pyvips:Inited libvips
>>> Image.new_from_file("/root/lib/test.png")
DEBUG:pyvips.voperation:VipsOperation.call: operation_name = VipsForeignLoadPngFile
DEBUG:pyvips.voperation:VipsOperation.call: match_image = None
DEBUG:pyvips.vobject:VipsObject.set: name = filename, value = /root/lib/test.png
DEBUG:pyvips.vobject:VipsObject.get: name = out
DEBUG:pyvips.vobject:VipsObject.get: name = interpretation
DEBUG:pyvips.vobject:VipsObject.get: name = width
DEBUG:pyvips.vobject:VipsObject.get: name = height
DEBUG:pyvips.vobject:VipsObject.get: name = format
DEBUG:pyvips.vobject:VipsObject.get: name = bands
DEBUG:pyvips.vobject:VipsObject.get: name = interpretation
DEBUG:pyvips.voperation:VipsOperation.call: result = <pyvips.Image 1240x1754 uchar, 1 bands, b-w>
DEBUG:pyvips.vobject:VipsObject.get: name = interpretation
DEBUG:pyvips.vobject:VipsObject.get: name = width
DEBUG:pyvips.vobject:VipsObject.get: name = height
DEBUG:pyvips.vobject:VipsObject.get: name = format
DEBUG:pyvips.vobject:VipsObject.get: name = bands
DEBUG:pyvips.vobject:VipsObject.get: name = interpretation
<pyvips.Image 1240x1754 uchar, 1 bands, b-w>
>>> Image.thumbnail_buffer(b"test", 128)
DEBUG:pyvips.voperation:VipsOperation.call: operation_name = thumbnail_buffer
DEBUG:pyvips.voperation:VipsOperation.call: match_image = None
DEBUG:pyvips.vobject:VipsObject.set: name = buffer, value = b'test'
DEBUG:pyvips.vobject:VipsObject.set: name = width, value = 128
INFO:pyvips:VIPS: thumbnailing 4 bytes of data
DEBUG:pyvips.error:Error unable to call thumbnail_buffer VipsForeignLoad: buffer is not in a known format

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/pyvips/vimage.py", line 256, in call_function
    return pyvips.Operation.call(name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pyvips/voperation.py", line 305, in call
    raise Error('unable to call {0}'.format(operation_name))
pyvips.error.Error: unable to call thumbnail_buffer
  VipsForeignLoad: buffer is not in a known format

but with a prebuilt libvips binary we'd be stuck in ABI mode

I installed pyvips with the prebuilt libvips-dev from kleisauke/libvips-packaging (with proper LD_LIBRARY_PATH and PKG_CONFIG_PATH set) and pip show reports the installation is using API mode. So I guess this might not be true?

You should just need the libvips-dev package

Actually the package manager will download a LOT of dependencies (over 100 using conda, over 200! using apt, including libjpg, libpng etc.), taking up more disk spaces than statically linked libvips-dev package, and also taking more time to download and install. More importantly, package managers like apt usually don't come with newer versions of these libraries, so that would be another down side.

jcupitt commented 11 months ago

Huh strange, it ought to work. I'll try to make a dockerfile for this.

Actually the package manager will download a LOT of dependencies (over 100 using conda, over 200! using apt, including libjpg, libpng etc.),

Did you use --no-install-recommends? The deb package recommends the libvips GUI, and that in turn pulls in most of X11 and gtk2 (!!).

jcupitt commented 11 months ago

I installed pyvips with the prebuilt libvips-dev from kleisauke/libvips-packaging (with proper LD_LIBRARY_PATH and PKG_CONFIG_PATH set) and pip show reports the installation is using API mode. So I guess this might not be true?

Ah if you use the libvips-dev precompiled binary it'd work, yes. But that's probably no smaller than just installing libvips-dev from your package manager, and will be annoying to maintain.

jcupitt commented 11 months ago

I updated this dockerfile:

https://github.com/jcupitt/docker-builds/blob/master/pyvips-alpine/Dockerfile

And it seems to work fine. That's libvips 8.15.1 and pyvips 2.2.1 on alpine.

I've not tried with those precompiled libvips binaries. If you need a newer libvips than is in your package manager, I'd expect building from source yourself to be much more likely to work, should be smaller (it can use the platform libjpeg etc.) and can be deployed pretty easily with docker or whatever.

SteveHawk commented 11 months ago

Did you use --no-install-recommends

Yes i did, in a python:3.11-slim-bullseye docker, run apt install --no-install-recommends libvips-dev resulted in:

1 upgraded, 260 newly installed, 0 to remove and 15 not upgraded.
Need to get 117 MB of archives.
After this operation, 475 MB of additional disk space will be used.

Without the --no-install-recommends flag that would be even more madness:

1 upgraded, 419 newly installed, 0 to remove and 15 not upgraded.
Need to get 260 MB of archives.
After this operation, 1002 MB of additional disk space will be used.

But that's probably no smaller than just installing libvips-dev from your package manager

The precompiled libvips-dev package is a 8MB tarball and only 26MB uncompressed on disk. I believe it only includes necessary .so files, so it's actually much smaller in size ;)

kleisauke commented 11 months ago

The prebuilt binaries available at https://github.com/kleisauke/libvips-packaging would only work with pyvips' API mode since GLib is statically-linked.

Here's a Dockerfile to reproduce this issue:

Details ```dockerfile # Build and run with: # docker build -t pyvips-libvips-packaging . # docker run -it --rm pyvips-libvips-packaging FROM python:3.12-slim-bookworm LABEL maintainer="Kleis Auke Wolthuizen " ARG VIPS_VERSION=8.15.0 # https://gist.github.com/kleisauke/91ef7315adbe1ca7455a38b24d28081e ARG VIPS_PC_FILE=https://gist.github.com/kleisauke/91ef7315adbe1ca7455a38b24d28081e/raw/8b780945bacb66d7e79efa1ecc1c9a783b2d1f47/vips.pc WORKDIR /opt/libvips # Update packages RUN apt-get update -q \ # Install needed dependencies && apt-get install -qy --no-install-recommends \ build-essential \ ca-certificates \ curl \ pkg-config \ # Download pre-built libvips binaries && curl -Ls https://github.com/kleisauke/libvips-packaging/releases/download/v$VIPS_VERSION/libvips-$VIPS_VERSION-linux-x64.tar.gz | tar -xzC . \ # Download pkgconfig file && mkdir -p lib/pkgconfig \ && cd lib/pkgconfig \ && curl -LO $VIPS_PC_FILE WORKDIR /app # Update the PKG_CONFIG_PATH environment variable, # since libvips is installed in a non-standard prefix ENV PKG_CONFIG_PATH=/opt/libvips/lib/pkgconfig \ # Ensure dynamic linker finds the pre-built libvips binaries LD_LIBRARY_PATH=/opt/libvips/lib RUN pip install --verbose --user pyvips \ && pip show pyvips CMD ["python"] ```

The salient part of the error log is:

DEBUG:pyvips:Binary module load failed: /root/.local/lib/python3.12/site-packages/_libvips.abi3.so: undefined symbol: vips_path_mode7
DEBUG:pyvips:Falling back to ABI mode
OSError: cannot load library 'libgobject-2.0.so.0': libgobject-2.0.so.0: cannot open shared object file: No such file or directory.  Additionally, ctypes.util.find_library() did not manage to locate a library called 'libgobject-2.0.so.0'

So, it tries to fallback to ABI mode since the deprecated vips_path_mode7 symbol is not available, this was fixed with commit https://github.com/libvips/pyvips/commit/350a4597ee06d846446fe9b847dde1952112e2ce.

jcupitt commented 11 months ago

Ah that had popped out of my brain, thanks Kleis.

Then I agree, a pyvips 2.2.2 would be useful. I'll do it now.

jcupitt commented 11 months ago

We might have v2.2.2, any testing very welcome.

kleisauke commented 11 months ago

Great! I just opened PR https://github.com/libvips/pyvips/pull/444 to support this in ABI mode.

FWIW, looks like v2.2.2 was miss-tagged as v2.2.0: https://github.com/libvips/pyvips/releases/tag/v2.2.0

kleisauke commented 11 months ago

pyvips still uses the prehistoric setup.py, so really the whole installer needs redoing and updating.

I'll have a look at this somewhere tomorrow.

jcupitt commented 11 months ago

FWIW, looks like v2.2.2 was miss-tagged as v2.2.0:

:facepalm: fixed.

SteveHawk commented 10 months ago

We might have v2.2.2, any testing very welcome.

Nice! It works well on my codes so far, will do more tests next week. Thanks!

kleisauke commented 10 months ago

pyvips still uses the prehistoric setup.py, so really the whole installer needs redoing and updating.

PR #445 should fix this.