search

python-pillow / Pillow

Python Imaging Library (Fork)
https://python-pillow.org
Other
12.32k stars 2.23k forks source link

Pillow 11.0.0 and Django: TiffImagePlugin raised 'Invalid dimensions' error #8530

Open jedie opened 2 weeks ago

jedie commented 2 weeks ago

Seems that v11.0.0 release add a Bug parsing tiff images:

...
  File "django/core/files/images.py", line 63, in get_image_dimensions
    p.feed(data)
  File "PIL/ImageFile.py", line 471, in feed
    im = Image.open(fp)
  File "PIL/Image.py", line 3515, in open
    im = _open_core(fp, filename, prefix, formats)
  File "PIL/Image.py", line 3503, in _open_core
    im = factory(fp, filename)
  File "PIL/TiffImagePlugin.py", line 1153, in __init__
    super().__init__(fp, filename)
  File "PIL/ImageFile.py", line 144, in __init__
    self._open()
  File "PIL/TiffImagePlugin.py", line 1177, in _open
    self._seek(0)
  File "PIL/TiffImagePlugin.py", line 1248, in _seek
    self._setup()
  File "PIL/TiffImagePlugin.py", line 1426, in _setup
    raise ValueError(msg)

more info in Sentry message:

grafik

Runtime: CPython v3.11.10(3.11.10 (main, Oct 19 2024, 01:04:28) [GCC 12.2.0])

It works with Pillow 10.4.0

radarhere commented 2 weeks ago

Could you attach a copy of the image?

jedie commented 2 weeks ago

no, sorry :( ...

radarhere commented 2 weeks ago

I'm guessing that you mean you can't for copyright/privacy reasons, but you still have a copy of the image you can test with? You can still replicate the problem on your end? Otherwise, you couldn't have determined that this passed with Pillow 10.4.0.

Are you sure that the image isn't actually invalid? Are you able to open the image using other software?

Are you able to modify your copy of Pillow to print xsize and ysize so we can see exactly how they are invalid?

If you're unable to upload the image, can you run exiftool -v -v over the image and paste the output here?

jedie commented 2 weeks ago

Yes, it's about copyright.

I can play with the image locally. It's very strange, i can't reproduce with:

import sys

import PIL
from PIL import Image

print(f'{sys.version=}')
print(f'{PIL.__version__=}')

with open("/tmp/foobar.tiff", "rb") as fp:
    im = Image.open(fp)
    print(f'{im.size=}')

It works in both cases:

sys.version='3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0]'
PIL.__version__='10.4.0'
im.size=(4000, 4000)
sys.version='3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0]'
PIL.__version__='11.0.0'
im.size=(4000, 4000)

exiftool:

  ExifToolVersion = 12.76
  FileName = foobar.tiff
  Directory = /tmp
  FileSize = 9666494
  FileModifyDate = 1730711096
  FileAccessDate = 1730711097
  FileInodeChangeDate = 1730711096
  FilePermissions = 33204
  FileType = TIFF
  FileTypeExtension = TIF
  MIMEType = image/tiff
  ExifByteOrder = II
  + [IFD0 directory with 22 entries]
  | 0)  ImageWidth = 4000
  |     - Tag 0x0100 (2 bytes, int16u[1])
  | 1)  ImageHeight = 4000
  |     - Tag 0x0101 (2 bytes, int16u[1])
  | 2)  BitsPerSample = 8 8 8
  |     - Tag 0x0102 (6 bytes, int16u[3])
  | 3)  Compression = 8
  |     - Tag 0x0103 (2 bytes, int16u[1])
  | 4)  PhotometricInterpretation = 2
  |     - Tag 0x0106 (2 bytes, int16u[1])
  | 5)  StripOffsets = 8 1460 2912 4488 5986 7407 8887 10362 11801 13256 14743 16231 17[snip]
  |     - Tag 0x0111 (16000 bytes, int32u[4000])
  | 6)  Orientation = 1
  |     - Tag 0x0112 (2 bytes, int16u[1])
  | 7)  SamplesPerPixel = 3
  |     - Tag 0x0115 (2 bytes, int16u[1])
  | 8)  RowsPerStrip = 1
  |     - Tag 0x0116 (2 bytes, int16u[1])
  | 9)  StripByteCounts = 1452 1452 1576 1498 1421 1480 1475 1439 1455 1487 1488 1402 1[snip]
  |     - Tag 0x0117 (16000 bytes, int32u[4000])
  | 10) XResolution = 300 (300/1)
  |     - Tag 0x011a (8 bytes, rational64u[1])
  | 11) YResolution = 300 (300/1)
  |     - Tag 0x011b (8 bytes, rational64u[1])
  | 12) PlanarConfiguration = 1
  |     - Tag 0x011c (2 bytes, int16u[1])
  | 13) ResolutionUnit = 2
  |     - Tag 0x0128 (2 bytes, int16u[1])
  | 14) Software = Adobe Photoshop Elements 6.0 Windows
  |     - Tag 0x0131 (37 bytes, string[37])
  | 15) ModifyDate = 2021:11:12 10:46:02
  |     - Tag 0x0132 (20 bytes, string[20])
  | 16) Predictor = 2
  |     - Tag 0x013d (2 bytes, int16u[1])
  | 17) ApplicationNotes (SubDirectory) -->
  |     - Tag 0x02bc (3670 bytes, undef[3670])
  | + [XMP directory, 3670 bytes]
  | | XMPToolkit = XMP Core 5.5.0
  | | Format = image/tiff
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/dc:format'
  | | ColorMode = 3
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/photoshop:ColorMode'
  | | History = 
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/photoshop:History'
  | | ICCProfileName = sRGB IEC61966-2.1
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/photoshop:ICCProfile'
  | | CreateDate = 2011-03-30T14:11:25+02:00
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmp:CreateDate'
  | | CreatorTool = Adobe Photoshop Elements 6.0 Windows
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmp:CreatorTool'
  | | MetadataDate = 2021-11-12T10:46:02+01:00
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmp:MetadataDate'
  | | ModifyDate = 2021-11-12T10:46:02+01:00
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmp:ModifyDate'
  | | DocumentID = xmp.did:C9172B3515206811994CDBA4105FCF75
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:DocumentID'
  | | InstanceID = uuid:8511B8538A5BE011A5A9C61357626A13
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:InstanceID'
  | | OriginalDocumentID = xmp.did:C9172B3515206811994CDBA4105FCF75
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:OriginalDocumentID'
  | | HistoryAction = created
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 10/xmpMM:action'
  | | HistoryInstanceID = xmp.iid:C9172B3515206811994CDBA4105FCF75
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 10/xmpMM:instanceID'
  | | HistorySoftwareAgent = Adobe Photoshop CS4 Macintosh
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 10/xmpMM:softwareAgent'
  | | HistoryWhen = 2011-03-30T14:11:25+02:00
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 10/xmpMM:when'
  | | HistoryAction = produced
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 11/xmpMM:action'
  | | HistorySoftwareAgent = Affinity Photo 1.10.1
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 11/xmpMM:softwareAgent'
  | | HistoryWhen = 2021-11-08T14:43:02+01:00
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 11/xmpMM:when'
  | | HistoryAction = produced
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 12/stEvt:action'
  | | HistorySoftwareAgent = Affinity Photo 1.10.1
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 12/stEvt:softwareAgent'
  | | HistoryWhen = 2021-11-12T10:46:02+01:00
  | | - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/xmpMM:History/rdf:Seq/rdf:li 12/stEvt:when'
  | 18) PhotoshopSettings (SubDirectory) -->
  |     - Tag 0x8649 (28 bytes, undef[28])
  | + [Photoshop directory, 28 bytes]
  | | IPTCDigest = .............B~
  | | - Tag 0x0425 (16 bytes)
  | 19) ExifOffset (SubDirectory) -->
  |     - Tag 0x8769 (4 bytes, int32u[1])
  | + [ExifIFD directory with 3 entries]
  | | 0)  ColorSpace = 1
  | |     - Tag 0xa001 (2 bytes, int16u[1])
  | | 1)  ExifImageWidth = 4000
  | |     - Tag 0xa002 (2 bytes, int16u[1])
  | | 2)  ExifImageHeight = 4000
  | |     - Tag 0xa003 (2 bytes, int16u[1])
  | 20) ICC_Profile (SubDirectory) -->
  |     - Tag 0x8773 (596 bytes, undef[596])
  | + [ICC_Profile directory with 11 entries, 596 bytes]
  | | ProfileHeader (SubDirectory) -->
  | | + [BinaryData directory, 128 bytes]
  | | | ProfileCMMType = lcms
  | | | - Tag 0x0004 (4 bytes, string[4])
  | | | ProfileVersion = 1072
  | | | - Tag 0x0008 (2 bytes, int16s[1])
  | | | ProfileClass = mntr
  | | | - Tag 0x000c (4 bytes, string[4])
  | | | ColorSpaceData = RGB 
  | | | - Tag 0x0010 (4 bytes, string[4])
  | | | ProfileConnectionSpace = XYZ 
  | | | - Tag 0x0014 (4 bytes, string[4])
  | | | ProfileDateTime = 2021 11 12 9 44 25
  | | | - Tag 0x0018 (12 bytes, int16u[6])
  | | | ProfileFileSignature = acsp
  | | | - Tag 0x0024 (4 bytes, string[4])
  | | | PrimaryPlatform = MSFT
  | | | - Tag 0x0028 (4 bytes, string[4])
  | | | CMMFlags = 0
  | | | - Tag 0x002c (4 bytes, int32u[1])
  | | | DeviceManufacturer = 
  | | | - Tag 0x0030 (4 bytes, string[4])
  | | | DeviceModel = 
  | | | - Tag 0x0034 (4 bytes, string[4])
  | | | DeviceAttributes = 0 0
  | | | - Tag 0x0038 (8 bytes, int32u[2])
  | | | RenderingIntent = 0
  | | | - Tag 0x0040 (4 bytes, int32u[1])
  | | | ConnectionSpaceIlluminant = 0.9642 1 0.82491
  | | | - Tag 0x0044 (12 bytes, fixed32s[3])
  | | | ProfileCreator = lcms
  | | | - Tag 0x0050 (4 bytes, string[4])
  | | | ProfileID = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  | | | - Tag 0x0054 (16 bytes, int8u[16])
  | | 0)  ProfileDescription = sRGB IEC61966-2.1
  | |     - Tag 'desc' (34 bytes, type 'mluc')
  | | 1)  ProfileCopyright = No copyright, use freely
  | |     - Tag 'cprt' (48 bytes, type 'mluc')
  | | 2)  MediaWhitePoint = 0.9642 1 0.82491
  | |     - Tag 'wtpt' (20 bytes, type 'XYZ ')
  | | 3)  ChromaticAdaptation = 1.04788 0.02292 -0.05022 0.02959 0.99048 -0.01707 -0.00[snip]
  | |     - Tag 'chad' (44 bytes, type 'sf32')
  | | 4)  RedMatrixColumn = 0.43604 0.22249 0.01392
  | |     - Tag 'rXYZ' (20 bytes, type 'XYZ ')
  | | 5)  BlueMatrixColumn = 0.14305 0.06061 0.71391
  | |     - Tag 'bXYZ' (20 bytes, type 'XYZ ')
  | | 6)  GreenMatrixColumn = 0.38512 0.7169 0.09706
  | |     - Tag 'gXYZ' (20 bytes, type 'XYZ ')
  | | 7)  RedTRC = para..ff...Y...[
  | |     - Tag 'rTRC' (32 bytes, type 'para')
  | | 8)  GreenTRC = para..ff...Y...[
  | |     - Tag 'gTRC' (32 bytes, type 'para')
  | | 9)  BlueTRC = para..ff...Y...[
  | |     - Tag 'bTRC' (32 bytes, type 'para')
  | | 10) Chromaticity (SubDirectory) -->
  | |     - Tag 'chrm' (36 bytes, type 'chrm')
  | | + [BinaryData directory, 36 bytes]
  | | | ChromaticityChannels = 3
  | | | - Tag 0x0008 (2 bytes, int16u[1])
  | | | ChromaticityColorant = 0
  | | | - Tag 0x000a (2 bytes, int16u[1])
  | | | ChromaticityChannel1 = 0.64 0.33
  | | | - Tag 0x000c (8 bytes, fixed32u[2])
  | | | ChromaticityChannel2 = 0.3 0.60001
  | | | - Tag 0x0014 (8 bytes, fixed32u[2])
  | | | ChromaticityChannel3 = 0.14999 0.06
  | | | - Tag 0x001c (8 bytes, fixed32u[2])
  | 21) Exif_0xc7e0 = 0 255 75 83 2 0 112 79 120 69 1 0 9 0 0 0 49 115 116 112 79 1 0 0[snip]
  |     - Tag 0xc7e0 (110 bytes, int8u[110])

But this runs on a different machine...

jedie commented 2 weeks ago

I also tried with Python 3.11:

sys.version='3.11.10 (main, Oct 16 2024, 04:23:21) [Clang 18.1.8 ]'
PIL.__version__='11.0.0'
im.size=(4000, 4000)

Strange. Any idea?

radarhere commented 2 weeks ago

If you're asking how Django could have manipulated the image before passing it to Pillow, then that sounds like a Django question more than a Pillow question.

If you're asking how to debug this further,

  1. I would suggest modifying either Django or Pillow locally to write the image data to disk at the exact point that it is passed to Pillow.
  2. If that then gives you an image that causes a problem without Django, then I suggest checking if that image opens in other software. If it doesn't, then it sounds like the image is corrupt, and not necessarily something we need to be opening successfully.
  3. If it does, then I suggest modifying Pillow to print xsize and ysize just before the ValueError is raised.
jedie commented 2 weeks ago

I can not say what's happening here... But i can say that's the combination between Django v4.2.16 and pillow '10.4.0' vs. '11.0.0'

>>> import django
>>> django.__version__
'4.2.16'
>>> Image.open((x.thumbnail.open('rb'))).width
4000
>>> import PIL
>>> PIL.__version__
'11.0.0'
>>> x.thumbnail.width
/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/TiffImagePlugin.py:935: UserWarning: Corrupt EXIF data.  Expecting to read 2 bytes but only got 0. 
  warnings.warn(str(msg))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/django/core/files/images.py", line 20, in width
    return self._get_image_dimensions()[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/django/core/files/images.py", line 30, in _get_image_dimensions
    self._dimensions_cache = get_image_dimensions(self, close=close)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/django/core/files/images.py", line 63, in get_image_dimensions
    p.feed(data)
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/ImageFile.py", line 471, in feed
    im = Image.open(fp)
         ^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/Image.py", line 3515, in open
    im = _open_core(fp, filename, prefix, formats)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/Image.py", line 3503, in _open_core
    im = factory(fp, filename)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/TiffImagePlugin.py", line 1153, in __init__
    super().__init__(fp, filename)
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/ImageFile.py", line 144, in __init__
    self._open()
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/TiffImagePlugin.py", line 1177, in _open
    self._seek(0)
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/TiffImagePlugin.py", line 1248, in _seek
    self._setup()
  File "/root/.local/share/virtualenvs/foobar/lib/python3.11/site-packages/PIL/TiffImagePlugin.py", line 1426, in _setup
    raise ValueError(msg)
ValueError: Invalid dimensions
>>>

Think it's related to the way how Django tries to get the image size: https://github.com/django/django/blob/968397228fe03968bb855856532569586c8a8a1c/django/core/files/images.py#L35-L89

jedie commented 2 weeks ago

Yes, it's the combination of how Django tries to get the image size via django.core.files.images.get_image_dimensions() here: https://github.com/django/django/blob/968397228fe03968bb855856532569586c8a8a1c/django/core/files/images.py#L35-L89 and this change: https://github.com/python-pillow/Pillow/commit/e6e5ef5c5fbd83ac5dd63301e4d7d6860a7b2d09#diff-6ad43f85f1a075181d4d8cfcd97ae27bba1eccf5c3db5a3457160f98218eba6eR1404 that added the raise ValueError:

        xsize = self.tag_v2.get(IMAGEWIDTH)
        ysize = self.tag_v2.get(IMAGELENGTH)
        if not isinstance(xsize, int) or not isinstance(ysize, int):
            msg = "Invalid dimensions"
            raise ValueError(msg)
jedie commented 2 weeks ago

I created https://code.djangoproject.com/ticket/35884 and https://github.com/django/django/pull/18764

sarahboyce commented 2 weeks ago

Hello :wave: Django Fellow here

In Django we are using Pillow as documented to parse an image: https://pillow.readthedocs.io/en/stable/reference/ImageFile.html#example-parse-an-image

Using this this file python-logo.tiff and following the example

from PIL import ImageFile

fp = open("python-logo.tiff", "rb")

p = ImageFile.Parser()

while 1:
    s = fp.read(1024)
    if not s:
        break
    p.feed(s)

im = p.close()

im.save("copy.tiff")

This raises the ValueError: Invalid dimensions error on Pillow 11

Traceback (most recent call last):
  File "/project-path/./main.py", line 11, in <module>
    p.feed(s)
  File "/project-path/.venv/lib/python3.10/site-packages/PIL/ImageFile.py", line 471, in feed
    im = Image.open(fp)
  File "/project-path/.venv/lib/python3.10/site-packages/PIL/Image.py", line 3520, in open
    im = _open_core(
  File "/project-path/.venv/lib/python3.10/site-packages/PIL/Image.py", line 3503, in _open_core
    im = factory(fp, filename)
  File "/project-path/.venv/lib/python3.10/site-packages/PIL/TiffImagePlugin.py", line 1153, in __init__
    super().__init__(fp, filename)
  File "/project-path/.venv/lib/python3.10/site-packages/PIL/ImageFile.py", line 144, in __init__
    self._open()
  File "/project-path/.venv/lib/python3.10/site-packages/PIL/TiffImagePlugin.py", line 1177, in _open
    self._seek(0)
  File "/project-path/.venv/lib/python3.10/site-packages/PIL/TiffImagePlugin.py", line 1248, in _seek
    self._setup()
  File "/project-path/.venv/lib/python3.10/site-packages/PIL/TiffImagePlugin.py", line 1426, in _setup
    raise ValueError(msg)
ValueError: Invalid dimensions

No error is raised on Pillow 10.4.0

We can add new error handling in Django but before doing so, I would like to confirm that this is not an issue in Pillow 11 and that the documented example is missing error handling also

radarhere commented 2 weeks ago

Hi. Thanks for breaking the problem down. The code change that led to this wasn't intended to change behaviour, and your use case is reasonable.

I've created #8535 to resolve this.