Closed jax-b closed 1 day ago
Hi. Would it be possible for you to put together some example code that only involves Pillow (not fastapi)? Two examples, one showing your opening problem, and the other showing your saving problem.
Also, your traceback does actually show the error itself. What is it?
i mean its only image open and image save to a BytesIO buffer. This just has the fastapi code around it to give better context. The traceback show that the it looks like the WebP handler is not registered within Pillow? its throwing a UnidentifiedImageError
Hi. Would it be possible for you to put together some example code that only involves Pillow (not fastapi)? Two examples, one showing your opening problem, and the other showing your saving problem.
Well, it is clear from the report that webp is not installed correctly with Pillow 10.4.0:
/opt/python/PIL/features.py:42: UserWarning: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/python/PIL/../pillow.libs/libsharpyuv-898c0cb5.so.0.1.0) warnings.warn(str(ex)) *** WEBP support not installed
So the question really is, @jax-b how did you install Pillow 10.4.0?
Looking at the filename, it seems you somehow managed to install pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64.whl
on a system with glibc<2.27 (edit: Amazon Linux 2 has glibc 2.26, so clearly older than 2.28), which should not be possible.
$ find . -iname libsharpyuv\*
./pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-898c0cb5.so.0.1.0
./pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64/pillow.libs/libsharpyuv-aaa00b5c.so.0.1.0
The pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
wheel that would be correct for your system does not require glibc 2.27:
$ ldd -v pillow.libs/libsharpyuv-aaa00b5c.so.0.1.0
linux-vdso.so.1 (0x00007ffe557fe000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000792d8318d000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x0000792d83188000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000792d82e00000)
/lib64/ld-linux-x86-64.so.2 (0x0000792d8328e000)
Version information:
pillow.libs/libsharpyuv-aaa00b5c.so.0.1.0:
libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libm.so.6:
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_2.36) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libpthread.so.0:
libc.so.6 (GLIBC_2.36) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6:
ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
both were installed as part of the serverless python requirements for lambda library. which then zip up the requirements for lambda and uploads it to AWS. it installs the modules to a custom target directory using a standard pip command
python -m pip install -r requirements.txt -t /tmp/dir
the requirements file is mostly the same just pinned at different versions of pillow
pillow==10.X.0
webp
the opening problem is fixed by the intermediary buffer so I wouldn't worry about that too much (it works between versions).
from PIL.WebPImagePlugin import Image
from io import BytesIO
test_webp: bytes = None
with open("test.webp", "rb") as f:
test_webp = f.read()
img = Image.open(BytesIO(test_webp))
full_size_bytes = BytesIO()
img.save(full_size_bytes, format="webp")
with open("test_out.webp", "wb") as f:
f.write(full_size_bytes)
if it works on one version but not another using the same base docker container i dont think that the issue would be the platform? Could how Pillow looks for the GLIBC version have changed or was the minimum version of GLIBC changed between 10.3.0 and 10.4.0.
my build system is WSL Ubuntu 22.04
Linux wsl 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
Something to note is looking at the wheel that works on Lambda its installing the cp310-cp310-manylinux_2_28_x86_64
version and it has no problems running?
You can try the --platform manylinux_2_17_x86_64
pip parameter to force it to download only compatible wheels as I suggested in https://github.com/python-pillow/Pillow/issues/7648#issuecomment-1871617284:
python -m pip install --platform manylinux_2_17_x86_64 --only-binary :all: -r requirements.txt -t /tmp/dir
Something to note is looking at the wheel that works on Lambda its installing the
cp310-cp310-manylinux_2_28_x86_64
version and it has no problems running?
Indeed, it looks like the version of webp included in the 10.3.0 wheel only needs glibc 2.14:
$ ldd -v pillow-10.3.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-20f78091.so.0.0.1
linux-vdso.so.1 (0x00007ffdb2957000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007de977813000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007de97780e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007de977400000)
/lib64/ld-linux-x86-64.so.2 (0x00007de977914000)
Version information:
pillow-10.3.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-20f78091.so.0.0.1:
libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libm.so.6:
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_2.36) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libpthread.so.0:
libc.so.6 (GLIBC_2.36) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6:
ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
Could how Pillow looks for the GLIBC version have changed or was the minimum version of GLIBC changed between 10.3.0 and 10.4.0.
The minimum glibc version has not changed, that remains to be 2.17 for the manylinux_2_17_x86_64
wheels and 2.28 for the manylinux_2_28_x86_64
wheels. However, just because that is the requirement, it does not mean that it definitely won't work on older glibc, just that it is not supported.
The included version of webp was updated from 1.3.2 to 1.4.0 in https://github.com/python-pillow/Pillow/pull/7973.
The new version now uses the logf
and expf
glibc functions, which were updated in glibc 2.27.
As you can see, compiling on a glibc 2.28 system creates a dependency on the new version of these functions:
$ objdump -T pillow-10.3.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-20f78091.so.0.0.1 | grep -E 'logf|expf'
$ objdump -T pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-898c0cb5.so.0.1.0 | grep -E 'logf|expf'
0000000000000000 DF *UND* 0000000000000000 (GLIBC_2.27) expf
0000000000000000 DF *UND* 0000000000000000 (GLIBC_2.27) logf
$ objdump -T pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64/pillow.libs/libsharpyuv-aaa00b5c.so.0.1.0 | grep -E 'logf|expf'
0000000000000000 DF *UND* 0000000000000000 (GLIBC_2.2.5) expf
0000000000000000 DF *UND* 0000000000000000 (GLIBC_2.2.5) logf
If it is helpful, here's a Pillow wheel built with libwebp 1.3.2 - pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.zip
That solves the issue
What did you do?
Installed the latest version of 10.4.0 to my AWS lambda test environment
What did you expect to happen?
No issues opening and saving webp files
What actually happened?
WebP files failed to open without changing to a intermediary BytesIO After changing to a intermediary BytesIO i still had trouble saving the image
Reverting Pillow to 10.3.0 restores operation
What are your OS, Python and Pillow versions?
pil features
10.4.0
10.3.0
Test Code
Traceback
Traceback is from my cloudwatch logs which use Mangum as a intermediary to run the API