search

sagemath / sage

Main repository of SageMath. Now open for Issues and Pull Requests.
https://www.sagemath.org
Other
1.21k stars 419 forks source link

sage containers missing PIL(low) jpeg support #37304

Open stdweird opened 5 months ago

stdweird commented 5 months ago

Steps To Reproduce

import PIL.Image
v=PIL.Image.open('sample_640_426.jpeg')
v.convert('L')

you get a backtrace that ends with

...
File /home/sage/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/PIL/Image.py:393, in _getdecoder(mode, decoder_name, args, extra)
    391 except AttributeError as e:
    392     msg = f"decoder {decoder_name} not available"
--> 393     raise OSError(msg) from e
    394 return decoder(mode, *args + extra)

OSError: decoder jpeg not available

searching about this issue suggests that libjpeg-dev is missing when pillow was being built.

adding libjpeg-dev is not enough, you must recompile PIL

Expected Behavior

that you can use jpeg images with PIL in SAGE

Actual Behavior

stacktrace due to missing OS libraries (at compiler, but ofcourse also at runtime).

Additional Information

I don't know the gitrepo enough to know where all OS dependencies are installed; i assume opening a PR to add it to the main Dockerfile is not an option

Environment

- **OS**: Ubuntu 22.04.3 LTS
- **Sage Version**: 10.2

From the dockerhub container

Checklist

soehms commented 5 months ago

I don't think this is a bug! According to the Sage reference manual libjpeg is classified as an optional package. The images in our Docker Hub repositories only contain standard packages (in order not to explode their size). You may create your own image derivative containing an optional package using the repository sagemath-dev for example using a dockerfile like this:

FROM sagemath/sagemath-dev:latest
RUN sudo apt-get -qq update && sudo apt-get -qq install -y --no-install-recommends libjpeg-dev && ./sage -pip install --no-cache-dir -I pillow
...

Maybe we should make this more clear in the readme file on Docker Hub.

I hope you understand that as causa libjpeg I don't want to make an exception. On the other hand, I have to confess that I don't know the reason why libjpeg is optional. I suspect this is because it is not available on all Sage supported systems. According to the build scripts it will be used whenever it is available on the system.

@mkoeppe , @saraedum, @fchapoton Would it make sense to pre-install everything that appears for Ubuntu at the end of .configure in the Docker Hub images? But, this would probably increase the size of the images significantly.

mkoeppe commented 5 months ago

Some of the "optional packages" are distinguished as "recommended"; libjpeg is one of them. See https://github.com/sagemath/sage/blob/develop/build/pkgs/_recommended/dependencies and https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#linux-system-package-installation It may be worth considering to add them, but container image size is of course an important consideration.

soehms commented 5 months ago

It may be worth considering to add them, but container image size is of course an important consideration.

Thanks for the hints! I will check locally first using build/pkgs/_recommended/dependencies how this effects the container image size.

stdweird commented 5 months ago

@soehms i understand if you don't want to add them. a short entry in a readme somewhere to add the very simple instructions is also fine for me.

wrt size, libjpeg-dev is not very large, esp not compared to the texlive code (that is huge). the current container is already 4GB or so, libjeg-dev will add less then a MB (texlive is 500-800MB).

the way we use it, we don't really care about the size. feel free to make a sage-full container that is 10GB that has all you can get, that is also ok for our use case.

and thanks for considering this.

soehms commented 5 months ago

@soehms i understand if you don't want to add them. a short entry in a readme somewhere to add the very simple instructions is also fine for me.

wrt size, libjpeg-dev is not very large, esp not compared to the texlive code (that is huge). the current container is already 4GB or so, libjeg-dev will add less then a MB (texlive is 500-800MB).

the way we use it, we don't really care about the size. feel free to make a sage-full container that is 10GB that has all you can get, that is also ok for our use case.

and thanks for considering this.

In the meantime I verified that the Dockerfile I suggested in https://github.com/sagemath/sage/issues/37304#issuecomment-1944291850 leads to an image that fixes your issue and has about 9.9GB (not much more as sagemath/sagemath-dev:latest where it is derived from). Another option for you would by to use our CI related images for example that for Ubuntu Jammy. Here libjpeg is available, too, and the image consumes about 9.6GB.

soehms commented 5 months ago

It may be worth considering to add them, but container image size is of course an important consideration.

Thanks for the hints! I will check locally first using build/pkgs/_recommended/dependencies how this effects the container image size.

Unfortunately this wasn't successful. After the following change on Dockerfile I got 11.8 GB for the make-build target.

git diff
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 0ebfe80961..cdc9f855d7 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -91,6 +91,7 @@ RUN ln -s /usr/bin/sage /usr/bin/sagemath
 # We also install sudo for the sage user, see below.
 RUN apt-get -qq update \
     && apt-get -qq install -y --no-install-recommends gfortran gcc g++ libstdc++-10-dev sudo openssl \
+                                   pandoc ffmpeg imagemagick git libjpeg-dev texlive texlive-luatex fonts-freefont-otf xindy \
     && apt-get -qq clean \
     && rm -r /var/lib/apt/lists/*

This is nearly 2 GB more than currently. But unfortunately this doesn't solve the issue. Even though libjpeg was successfully installed it was not recovered be configure:

Selecting previously unselected package libjpeg-turbo8:amd64.
Preparing to unpack .../017-libjpeg-turbo8_2.1.2-0ubuntu1_amd64.deb ...
Unpacking libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ...
Selecting previously unselected package libjpeg8:amd64.
Preparing to unpack .../018-libjpeg8_8c-2ubuntu10_amd64.deb ...
Unpacking libjpeg8:amd64 (8c-2ubuntu10) ...
Selecting previously unselected package libjpeg-turbo8-dev:amd64.
Preparing to unpack .../264-libjpeg-turbo8-dev_2.1.2-0ubuntu1_amd64.deb ...
Unpacking libjpeg-turbo8-dev:amd64 (2.1.2-0ubuntu1) ...
Selecting previously unselected package libjpeg8-dev:amd64.
Preparing to unpack .../265-libjpeg8-dev_8c-2ubuntu10_amd64.deb ...
Unpacking libjpeg8-dev:amd64 (8c-2ubuntu10) ...
Selecting previously unselected package libjpeg-dev:amd64.
Preparing to unpack .../266-libjpeg-dev_8c-2ubuntu10_amd64.deb ...
Unpacking libjpeg-dev:amd64 (8c-2ubuntu10) ...
Setting up libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ...
Setting up libjpeg8:amd64 (8c-2ubuntu10) ...
Setting up libjpeg-turbo8-dev:amd64 (2.1.2-0ubuntu1) ...
Setting up libjpeg8-dev:amd64 (8c-2ubuntu10) ...
Setting up libjpeg-dev:amd64 (8c-2ubuntu10) ...
....
Checking whether SageMath should install SPKG libjpeg...
checking for libjpeg... no
configure: no suitable system package found for SPKG libjpeg
....
libjpeg:                        no suitable system package; optional
....
[pillow-10.1.0]   Package libjpeg was not found in the pkg-config search path.
[pillow-10.1.0]   Perhaps you should add the directory containing `libjpeg.pc'
[pillow-10.1.0]   Package 'libjpeg', required by 'virtual:world', not found
[pillow-10.1.0]   Looking for `libjpeg` using pkg-config.
[pillow-10.1.0]   Package libjpeg was not found in the pkg-config search path.
[pillow-10.1.0]   Perhaps you should add the directory containing `libjpeg.pc'
[pillow-10.1.0]   Package 'libjpeg', required by 'virtual:world', not found
[pillow-10.1.0]   Looking for `libjpeg` using pkg-config.

On the other hand other packages I've added like pandoc or texlive-luatex have been found on the sytsem.

@mkoeppe Any idea what I did wrong?

But, if you don't have a quick answer it's probably not worth it to spend too much time with it. Is there something standing against pointing in the readme on Docker Hub to our CI images on ghcr.io as another option?

mkoeppe commented 5 months ago

Do you have a config.log for me?

soehms commented 4 months ago

Do you have a config.log for me?

See 37304_config.log.

I've also saved the console-output of docker build. If you need parts of it, please let me know!