Closed SnorrenEnzo closed 3 years ago
Do you have an example file you could attach that reproduces the problem?
You said it worked flawlessly before. Which Pillow version that was with?
The file is larger than 10 MB, so here is a download link: https://mega.nz/file/V9cGkD7C#Bi-M5H4X9tyv4sPmceJ2lc3bELPiSBECrYe8QsKFAI4
I do not recall in what version of Pillow it was working previously.
Thanks, I can repro with Pillow 8.1.0 and with the preceding 8.0.1 get this:
<PIL.TiffImagePlugin.TiffImageFile image mode=RGB size=6000x4000 at 0x10E917DF0>
OJPEGDecodeRaw: Inconsistent number of MCU in codestream.
I'm pretty surprised that a CR2 image actually works with Pillow at all, given that it's camera raw and Pillow hasn't ever tried to support it. On the other hand, lots of file formats are actually Tiff under the hood, so it's possible to get metadata from them even if we don't understand the actual pixels.
git bisect says eb8c1206d6b170d4e798a00db7432e023853da5c is the first bad commit, from PR https://github.com/python-pillow/Pillow/pull/5175 "Fix TIFF OOB Write error".
And opening the image in Mac Preview looks like this:
Yeah, CoreImage/Quartz has been able to do raw processing since forever. 10.4 at least IIRC.
img.show()
works with Pillow version 8.0.1 on this test image and several other .CR2 images. This is a good temporary fix for me for now.
It may be a little different, but it's a similar issue. https://stackoverflow.com/questions/64683811/how-to-debug-oserror-9-saving-a-tiff-via-libtiff-and-pillow
CoreImage/Quartz has been able to do raw processing since forever
It might not be actual raw processing. It might be displaying just the embedded JPEG or preview image, which I guess is what Pillow also tries to access.
There are multiple images in this TIFF file, the actual "raw" is in IFD3 (though Canon sRAW is not really raw data and is to be avoided). IFD0 is a 3 color channel (i.e. already processed RGB) 3.5 MB JPEG:
---- ExifTool ----
ExifToolVersion : 12.16
Warning : [minor] Missing 2 entries in CanonCustom2 group 4 directory
Warning : Possibly corrupted CanonCustom2 data
---- System ----
FileName : img.CR2
Directory : .
FileSize : 29 MiB
FileModifyDate : 2021:03:09 09:02:56+01:00
FileAccessDate : 2021:03:09 09:03:37+01:00
FileCreateDate : 2021:03:09 09:02:55+01:00
FilePermissions : rw-rw-rw-
---- File ----
FileType : CR2
FileTypeExtension : cr2
MIMEType : image/x-canon-cr2
ExifByteOrder : Little-endian (Intel, II)
---- IFD0 ----
ImageWidth : 6000
ImageHeight : 4000
BitsPerSample : 8 8 8
Compression : JPEG (old-style)
ImageDescription :
Make : Canon
Model : Canon EOS M6
PreviewImageStart : 26490288
Orientation : Horizontal (normal)
PreviewImageLength : 3654910
XResolution : 180
YResolution : 180
ResolutionUnit : inches
ModifyDate : 2021:02:14 17:20:05
Artist : Jelle Mes
Copyright : Jelle Mes
PreviewImage : (Binary data 3654910 bytes, use -b option to extract)
---- ExifIFD ----
ExposureTime : 1/250
FNumber : 10.0
ExposureProgram : Manual
ISO : 250
SensitivityType : Recommended Exposure Index
RecommendedExposureIndex : 250
ExifVersion : 0230
DateTimeOriginal : 2021:02:14 17:20:05
CreateDate : 2021:02:14 17:20:05
ComponentsConfiguration : Y, Cb, Cr, -
CompressedBitsPerPixel : 3
ShutterSpeedValue : 1/251
ApertureValue : 10.0
ExposureCompensation : 0
MeteringMode : Multi-segment
Flash : Off, Did not fire
FocalLength : 200.0 mm
UserComment :
SubSecTime : 92
SubSecTimeOriginal : 92
SubSecTimeDigitized : 92
FlashpixVersion : 0100
ColorSpace : sRGB
ExifImageWidth : 6000
ExifImageHeight : 4000
FocalPlaneXResolution : 6825.938567
FocalPlaneYResolution : 6825.938567
FocalPlaneResolutionUnit : inches
SensingMethod : One-chip color area
FileSource : Digital Camera
CustomRendered : Normal
ExposureMode : Manual
WhiteBalance : Auto
DigitalZoomRatio : 1
SceneCaptureType : Standard
OwnerName :
SerialNumber : 423050000229
LensInfo : 55-200mm f/0
LensModel : EF-M55-200mm f/4.5-6.3 IS STM
LensSerialNumber : 000002b9f0
---- Canon ----
MacroMode : Normal
SelfTimer : Off
Quality : RAW
CanonFlashMode : Off
ContinuousDrive : Single
FocusMode : AF + MF
RecordMode : CR2
CanonImageSize : n/a
EasyMode : Manual
DigitalZoom : None
Contrast : Normal
Saturation : Normal
CameraISO : 250
MeteringMode : Evaluative
FocusRange : Auto
AFPoint : Face Detect
CanonExposureMode : Manual
LensType : Canon EF-M 55-200mm f/4.5-6.3 IS STM
MaxFocalLength : 200 mm
MinFocalLength : 55 mm
FocalUnits : 1/mm
MaxAperture : 6.3
MinAperture : 32
FlashActivity : 0
FlashBits : (none)
FocusContinuous : Continuous
ImageStabilization : On (2)
DisplayAperture : 17
ZoomSourceWidth : 6000
ZoomTargetWidth : 6000
SpotMeteringMode : Center
ManualFlashOutput : n/a
ColorTone : Normal
SRAWQuality : n/a
CanonFlashInfo : 0 0 0 0
AutoISO : 100
BaseISO : 248
MeasuredEV : 12.41
TargetAperture : 10
TargetExposureTime : 1/251
ExposureCompensation : 0
WhiteBalance : Auto
SlowShutter : Off
SequenceNumber : 0
OpticalZoomCode : 0
FlashGuideNumber : 0
FlashExposureComp : 0
AutoExposureBracketing : Off
AEBBracketValue : 0
ControlMode : Camera Local Control
FNumber : 10
ExposureTime : 1/256
BulbDuration : 0
CameraType : Compact
AutoRotate : None
NDFilter : n/a
SelfTimer2 : 0
CanonImageType : IMG:High definition image
CanonFirmwareVersion : Firmware Version 1.00
FileNumber : 116-0146
CameraTemperature : 212 C
CanonModelID : EOS M6
ThumbnailImageValidArea : 0 159 7 112
Canon_0x0018 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0[...]
Canon_0x0019 : 1
DateStampMode : Off
FirmwareRevision : 1.00 rev 2.00
Categories : (none)
IntelligentContrast : n/a
ImageUniqueID : 041cb08c89853189680b56b9e5e6d06b
Canon_0x002d : 0
Canon_0x002e : 70 32767 32767 65535 65535 32767 32767 32767 60 65535 4[...]
FacesDetected : 65535
Canon_0x0031 : 12 1 65535 65535 65535 65535
Canon_0x0033 : 993809222 2301707739 3109423976 1808852709
TimeZone : +01:00
TimeZoneCity : (not set)
DaylightSavings : Off
Canon_0x0036 : 0 159 7 112 1620 1080
Canon_0x0040 : 8 0
Canon_0x0038 : ...½<
AFInfoSize : 436
AFAreaMode : Unknown (15)
NumAFPoints : 49
ValidAFPoints : 9
CanonImageWidth : 6000
CanonImageHeight : 4000
AFImageWidth : 6000
AFImageHeight : 4000
AFAreaWidths : 684 684 684 684 684 684 684 684 684 -1 0 -1 0 -28380 -1 -1 -1 0 -1 0 -28380 -1 0 -4558 -4558 1 -4558 -30084 -29828 -4558 0 -29828 4109 0 -29836 0 -24092 0 -29744 536 0 1877 -29744 0 0 0 19124 0 8008
AFAreaHeights : 420 420 420 420 420 420 420 420 420 16412 -8641 -28881 103 24832 0 -24288 16412 -8641 -28881 103 24832 0 0 -1 -1 0 -1 4956 -29836 -1 4956 -29836 -31815 4109 -1 -29828 4956 0 4096 536 -30783 243 0 -1 51 -1 7539 5 0
AFAreaXPositions : -684 0 684 -684 0 684 -684 0 684 536 536 536 536 536 536 536 536 536 536 536 536 536 536 -30080 0 -1 0 -30148 8 -30166 8 -30148 0 0 -30166 -30128 0 -30080 684 684 684 684 684 684 684 684 684 -1 8
AFAreaYPositions : -420 -420 -420 0 0 0 420 420 420 536 536 536 536 536 536 536 536 536 536 536 536 536 536 4 31171 0 31679 8 -1 -30240 2 -30136 -30128 30297 4 -30240 4 2 420 420 420 420 420 420 420 420 420 8 15013
AFPointsInFocus : 6
AFPointsSelected : (none)
Canon_0x003d : 32 1 0 0 0 0 0 0
Canon_0x003f : 1
Canon_0x0041 : 4149 200 99 55 200 3208 51 88 61 61 255 1 136 0 63 4 31 0
BracketMode : Off
BracketValue : 0
BracketShotNumber : 0
RawJpgQuality : RAW
WBBracketMode : Off
WBBracketValueAB : 0
WBBracketValueGM : 0
LiveViewShooting : On
FocusDistanceUpper : 32.08 m
FocusDistanceLower : 23.49 m
FlashExposureLock : Off
InternalSerialNumber : CJ0010606
AspectRatio : 3:2
CroppedImageWidth : 6000
CroppedImageHeight : 4000
CroppedImageLeft : 0
CroppedImageTop : 0
ToneCurve : Standard
Sharpness : 3
SharpnessFrequency : Unknown (-1)
SensorRedLevel : -1
SensorBlueLevel : -1
WhiteBalanceRed : -1
WhiteBalanceBlue : -1
ColorTemperature : 5200
PictureStyle : Auto
DigitalGain : 0
WBShiftAB : 0
WBShiftGM : 0
MeasuredRGGB : 1798 1024 1024 1229
VRDOffset : 0
SensorWidth : 6288
SensorHeight : 4056
SensorLeftBorder : 276
SensorTopBorder : 48
SensorRightBorder : 6275
SensorBottomBorder : 4047
BlackMaskLeftBorder : 0
BlackMaskTopBorder : 0
BlackMaskRightBorder : 0
BlackMaskBottomBorder : 0
ColorDataVersion : -4 (M100/M5/M6)
WB_RGGBLevelsAsShot : 1695 1024 1024 1301
ColorTempAsShot : 6172
WB_RGGBLevelsAuto : 1695 1024 1024 1301
ColorTempAuto : 6172
WB_RGGBLevelsMeasured : 1695 1024 1024 1301
ColorTempMeasured : 6172
WB_RGGBLevelsUnknown : 1552 1024 1024 1330
ColorTempUnknown : 5496
WB_RGGBLevelsDaylight : 1552 1024 1024 1330
ColorTempDaylight : 5496
WB_RGGBLevelsShade : 1818 1024 1024 1151
ColorTempShade : 7438
WB_RGGBLevelsCloudy : 1695 1024 1024 1231
ColorTempCloudy : 6559
WB_RGGBLevelsTungsten : 1074 1024 1024 1954
ColorTempTungsten : 3137
WB_RGGBLevelsFluorescent : 1323 1024 1024 1766
ColorTempFluorescent : 3923
WB_RGGBLevelsKelvin : 1552 1024 1024 1330
ColorTempKelvin : 5499
WB_RGGBLevelsFlash : 1956 1024 1024 1064
ColorTempFlash : 8416
WB_RGGBLevelsUnknown2 : 1595 1024 1024 1364
ColorTempUnknown2 : 5502
WB_RGGBLevelsUnknown3 : 1595 1024 1024 1364
ColorTempUnknown3 : 5502
WB_RGGBLevelsUnknown4 : 1595 1024 1024 1364
ColorTempUnknown4 : 5502
WB_RGGBLevelsUnknown5 : 1595 1024 1024 1364
ColorTempUnknown5 : 5502
WB_RGGBLevelsUnknown6 : 1595 1024 1024 1364
ColorTempUnknown6 : 5502
WB_RGGBLevelsUnknown7 : 1552 1024 1024 1330
ColorTempUnknown7 : 5496
WB_RGGBLevelsUnknown8 : 1552 1024 1024 1330
ColorTempUnknown8 : 5496
WB_RGGBLevelsUnknown9 : 1552 1024 1024 1330
ColorTempUnknown9 : 5496
WB_RGGBLevelsUnknown10 : 1552 1024 1024 1330
ColorTempUnknown10 : 5496
WB_RGGBLevelsUnknown11 : 1552 1024 1024 1330
ColorTempUnknown11 : 5496
WB_RGGBLevelsUnknown12 : 2675 1024 1024 1476
ColorTempUnknown12 : 7825
WB_RGGBLevelsUnknown13 : 3399 1024 1024 1661
ColorTempUnknown13 : 8151
CameraColorCalibration01 : 0 0 518 1051 (10900K)
CameraColorCalibration02 : 0 0 530 1028 (10000K)
CameraColorCalibration03 : 0 0 560 967 (8300K)
CameraColorCalibration04 : 0 0 596 906 (7000K)
CameraColorCalibration05 : 0 0 638 847 (6000K)
CameraColorCalibration06 : 0 0 664 817 (5600K)
CameraColorCalibration07 : 0 0 695 785 (5200K)
CameraColorCalibration08 : 0 0 737 736 (4700K)
CameraColorCalibration09 : 0 0 785 682 (4200K)
CameraColorCalibration10 : 0 0 840 634 (3800K)
CameraColorCalibration11 : 0 0 897 596 (3500K)
CameraColorCalibration12 : 0 0 968 553 (3200K)
CameraColorCalibration13 : 0 0 1027 521 (3000K)
CameraColorCalibration14 : 0 0 1099 486 (2800K)
CameraColorCalibration15 : 0 0 1283 444 (2400K)
PerChannelBlackLevel : 512 512 512 512
NormalWhiteLevel : 13035
SpecularWhiteLevel : 14338
Flavor : (Binary data 23980 bytes, use -b option to extract)
PictureStyleUserDef : Auto; Auto; Auto
PictureStylePC : n/a; n/a; n/a
CustomPictureStyleFileName :
Canon_0x4012 :
VignettingCorrVersion : 0
PeripheralLighting : On
DistortionCorrection : Off
ChromaticAberrationCorr : On
ChromaticAberrationCorr : On
PeripheralLightingValue : 1
DistortionCorrectionValue : 0
OriginalImageWidth : 100
OriginalImageHeight : 1126
PeripheralLightingSetting : On
ChromaticAberrationSetting : On
DistortionCorrectionSetting : Off
PeripheralIlluminationCorr : Off
AutoLightingOptimizer : Standard
HighlightTonePriority : Off
LongExposureNoiseReduction : Off
HighISONoiseReduction : Standard
AmbienceSelection : Standard
Canon_0x4023 : 16 0 0 0
GrainyBWFilter : Off
SoftFocusFilter : Off
ToyCameraFilter : Off
MiniatureFilter : Off
MiniatureFilterOrientation : Unknown (-1)
MiniatureFilterPosition : -1
MiniatureFilterParameter : -1
FisheyeFilter : Off
PaintingFilter : Off
WatercolorFilter : Off
Canon_0x402b : 24 0 0 0 0 0
Canon_0x402c : 8 0
Canon_0x402e : ÿÿÿÿüÿïÿŒÿÿ?ÿÿÿÿûÿìÿ?ÿþ?.ðÿ{ÿûüró¢[...]
---- CanonCustom ----
ISOExpansion : Off
SafetyShift : Disable
DialDirectionTvAv : Normal
CustomControls : 0 0 1 1 0 45 11 0 42 44
---- InteropIFD ----
InteropIndex : R98 - DCF basic file (sRGB)
InteropVersion : 0100
RelatedImageWidth : 6000
RelatedImageHeight : 4000
---- IFD1 ----
ThumbnailOffset : 45980
ThumbnailLength : 12387
ThumbnailImage : (Binary data 12387 bytes, use -b option to extract)
---- IFD2 ----
ImageWidth : 600
ImageHeight : 400
BitsPerSample : 16 16 16
Compression : Uncompressed
PhotometricInterpretation : RGB
StripOffsets : 25050288
SamplesPerPixel : 3
RowsPerStrip : 400
StripByteCounts : 1440000
PlanarConfiguration : Chunky
Exif_0xc5d9 : 2
SRawType : 3
Exif_0xc6dc : 0 0 0 0
---- IFD3 ----
Compression : JPEG (old-style)
StripOffsets : 58368
StripByteCounts : 24991920
Exif_0xc5d8 : 1
CR2CFAPattern : [Red,Green][Green,Blue]
RawImageSegmentation : 0 0 6288
SRawType : 1
---- Composite ----
DriveMode : Single-frame Shooting
ISO : 250
Lens : 55.0 - 200.0 mm
ShootingMode : Manual
WB_RGGBLevels : 1695 1024 1024 1301
Aperture : 10.0
BlueBalance : 1.270508
ImageSize : 6000x4000
LensID : Canon EF-M 55-200mm f/4.5-6.3 IS STM
Megapixels : 24.0
RedBalance : 1.655273
ScaleFactor35efl : 1.6
ShutterSpeed : 1/250
SubSecCreateDate : 2021:02:14 17:20:05.92
SubSecDateTimeOriginal : 2021:02:14 17:20:05.92
SubSecModifyDate : 2021:02:14 17:20:05.92
Lens35efl : 55.0 - 200.0 mm (35 mm equivalent: 88.7 - 322.5 mm)
CircleOfConfusion : 0.019 mm
DOF : 7.26 m (24.62 - 31.88 m)
FOV : 6.4 deg
FocalLength35efl : 200.0 mm (35 mm equivalent: 322.5 mm)
HyperfocalDistance : 214.66 m
LightValue : 13.3
Another part of the puzzle might be that the mandatory SamplesPerPixel
is suspiciously missing in the IFD0 of these CR2 files, maybe there was a change relating to parsing of that tag?
For example, when BitsPerSample
has a count of 3 like here, one should check/set SamplesPerPixel
to be 3 instead of default 1 when missing?
It looks like missing SamplesPerPixel
(and PhotometricInterpretation
) is common w/ OJPEG compressed images, so that shouldn't be the root cause as the fallbacks seem to have been there for a while.
OTOH, there have been quite a few changes to decoding OJPEG images since libtiff 4.0.10 when the problem started appearing, so this might not be Pillow's fault... But at the same time, it seems it should be possible to make it work with libtiff 4.2.0.
Testing, I find the -9 error was caused by #5175 - that was fixed in #5372. But #5372 introduced a new error - the image can't be identified, as it has "unknown data organization" - the SAMPLESPERPIXEL tag is missing. #5452 resolves this.
I thought the OSError: -9 was coming somewhere later from the C code in this case, i.e. inside decoder.decode()
?
I've updated my comment - #5372 has C changes that fix the -9.
Hi everyone,
Today I wanted to run a script I regularly use for image sorting, however I got an
OSError: -9
occurring at theimg.show()
. A simplified script which can replicate this behaviour is provided at the end of this post. The image I was trying to show is a .CR2, a Canon RAW image. I have used this script extensively for more than a year on .CR2 files and it behaved flawlessly all the time, until now. It occurs with every .CR2 image, even ones that I know I have opened with Pillow before.The full error log:
Installed modules (made a fresh conda env for this report):
What did you do?
Run the script below.
What did you expect to happen?
To show the image.
What actually happened?
An error occurred and no image was shown.
What are your OS, Python and Pillow versions?
Code: