10% of photos fail to load in darkroom, header corrupt

iisharankov commented 7 months ago

Describe the bug

Certain images fail to load consistantly between multiple versions and OSs of darktable. They work and are viewable on other applications, and even in darktable in the lighttable view. But in darkroom the below error is given. From my hours of googling and debugging with no avail, the issue seems to be something that causes the header file to fail to be loaded, which causes the colour matrix not to be found and the camera not to be found in the database.

Steps to reproduce

Import files into darktable. All of them are imported successfully in lighttable and their thumbnails visible. Go to darkroom, some photos load, others do not and kick you to lighttable. It's unclear what causes a file to be corrupted or error prone to do this, but the files it does this to are consistent. I've dived into hex editors and on the surface nothing seemed wrong, but that was about the time i gave up and decided I needed help...

Expected behavior

darktable should be able to open the files in darkroom so i can edit them normally

Logfile | Screenshot | Screencast

When opening about 10% of my photos (.ORF type on a Olympus E-M5 Mark III) I get the following error:

Text: (for google searches if someone else stumbles on this)

"darktable could not load ‘IMAGE.ORF’ , switching to lighttable now. Olympus E-M5 Mark III color matrix not found for image

please check that the camera model that produced the image is supported in darktable (list of supported cameras is at https://www.darktable.org/resources/camera-support/). if you are sure that the camera model is supported, please consider opening an issue at https://github.com/darktable-org/darktable"

This error is repeatable with the same images every time i try to open them, nothing has been able to fix them. I've also gotten it on my Olympus E-M5 Mark II, so it's on multiple cameras. It effects between 5% and 10% of photos. I am able to open them in other raw GUI viewers and see them, and the header data is nearly identical between working and broken images spare semantics.

I am on Ubuntu but have also tried on a fresh install on a Windows PC with the same files, and the same ones are "corrupted", leading me to believe it's the file and how darktable reacts to them. But it makes no sense as many of these photos are taken seconds apart from each other from the exact same camera and setup. I ran darktable with -d opencl but don't think that's the related issue, but nevertheless here you go:

darktable -d opencl
darktable 4.7.0~git850.928f08d3-1+11510.1
Copyright (C) 2012-2024 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> ENABLED
  Lua                    -> ENABLED  - API version 9.3.0
  Colord                 -> ENABLED
  gPhoto2                -> ENABLED
  GMIC                   -> ENABLED  - Compressed LUTs are supported
  GraphicsMagick         -> ENABLED
  ImageMagick            -> DISABLED
  libavif                -> DISABLED
  libheif                -> DISABLED
  libjxl                 -> DISABLED
  OpenJPEG               -> ENABLED
  OpenEXR                -> ENABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

     0.0262 [dt_get_sysresource_level] switched to 1 as `default'
     0.0262   total mem:       31715MB
     0.0262   mipmap cache:    3964MB
     0.0262   available mem:   15857MB
     0.0262   singlebuff:      247MB
     0.0647 [opencl_init] opencl disabled via darktable preferences
     0.0648 [dt_dlopencl_init] could not find default opencl runtime library 'libOpenCL'
     0.0649 [dt_dlopencl_init] could not find default opencl runtime library 'libOpenCL.so'
     0.0650 [opencl_init] opencl library 'libOpenCL.so.1' found on your system and loaded, preference 'default path'
     0.0651 [opencl_init] 0 platforms detected, error: Unknown OpenCL error
     0.0651 [opencl_init] FINALLY: opencl is NOT AVAILABLE and NOT ENABLED.
    25.3948 [rawspeed] (A2.ORF) rawspeed::RawImage rawspeed::RawDecoder::decodeRaw(), line 337: rawspeed::Array1DRef<const unsigned char> rawspeed::BitStreamerForwardSequentialReplenisher<Tag>::getInput() [with Tag = rawspeed::BitStreamerMSB], line 122: Buffer overflow read in BitStreamer
    25.3949 [libraw_open] extensions whitelist: `cr3'
    25.3949 [qoi_open] no proper file header in /home/.../darktable/examples/A2.ORF
    25.3973 [temperature] `Olympus E-M5 Mark III' color matrix not found for image
    25.4023 [temperature] `Olympus E-M5 Mark III' color matrix not found for image
    25.4160 [colorin] could not find requested profile `standard color matrix'!
   183.9323 [rawspeed] (B2.ORF) rawspeed::RawImage rawspeed::RawDecoder::decodeRaw(), line 337: rawspeed::Array1DRef<const unsigned char> rawspeed::BitStreamerForwardSequentialReplenisher<Tag>::getInput() [with Tag = rawspeed::BitStreamerMSB], line 122: Buffer overflow read in BitStreamer
   183.9324 [libraw_open] extensions whitelist: `cr3'
   183.9324 [qoi_open] no proper file header in /home/.../darktable/examples/B2.ORF
   183.9348 [temperature] `Olympus E-M5 Mark III' color matrix not found for image
   183.9399 [temperature] `Olympus E-M5 Mark III' color matrix not found for image
   183.9539 [colorin] could not find requested profile `standard color matrix'!
   188.7609 [rawspeed] (C2.ORF) rawspeed::RawImage rawspeed::RawDecoder::decodeRaw(), line 337: rawspeed::Array1DRef<const unsigned char> rawspeed::BitStreamerForwardSequentialReplenisher<Tag>::getInput() [with Tag = rawspeed::BitStreamerMSB], line 122: Buffer overflow read in BitStreamer
   188.7610 [libraw_open] extensions whitelist: `cr3'
   188.7611 [qoi_open] no proper file header in /home/.../darktable/examples/C2.ORF
   188.7636 [temperature] `Olympus E-M5 Mark III' color matrix not found for image
   188.7689 [temperature] `Olympus E-M5 Mark III' color matrix not found for image
   188.7871 [colorin] could not find requested profile `standard color matrix'!

There seems to be a buffer overflow but I cannot deduce more as is.

Commit

No response

Where did you obtain darktable from?

distro packaging

darktable version

darktable 4.7.0~git850.928f08d3-1+11510.1

What OS are you using?

Linux

What is the version of your OS?

Ubuntu 22.04

Describe your system?

Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz
32GB Ram
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-H GT2 [UHD Graphics 630] (rev 02)
X11
ii libgtk-3-0:amd64, 3.24.33-1ubuntu2

Are you using OpenCL GPU in darktable?

No

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

Can you reproduce with another darktable version(s)?
- Yes, existed on earlier version while updating to try to fix. Existed in some version of 3.X as well but now I'm on the latest. Also replicated on a Windows machine with the latest darktable version/download.
Can you reproduce with a RAW or Jpeg or both?
- RAW-file-format, though I have not tried thoroughly with JPG
Are the steps above reproducible with a fresh edit (i.e. after discarding history)?
- yes, seems so
If the issue is with the output image, attach an XMP file if (you'll have to change the extension to .txt)
- No
Is the issue still present using an empty/new config-dir (e.g. start darktable with --configdir "/tmp")?
- Yes
Do you use lua scripts?
- No, none

Apologies for the size. I've given three working files and three broken ones. The _2.zip ones are the ones that are broken.

A1.zip A2.zip B1.zip B2.zip C1.zip C2.zip

LebedevRI commented 7 months ago

A2.zip

Looking at this one, it happens at row 3442 of 3912. If i comment out

https://github.com/darktable-org/rawspeed/blob/0d2b97f7e7f81b2a4c8ab5a1c834b7626da6b141/src/librawspeed/bitstreams/BitStreamer.h#L117-L119

i get: Are you 100% confident that the card you use in the camera isn't faulty?

LebedevRI commented 7 months ago

Looks like adobe dng converter also does not like these. I very strongly suspect that there's either a hardware issue (memory card/card reader/camera's card writer/cable), or bitrot after copying.

ralfbrown commented 7 months ago

The other applications which are showing the images without corruption are almost certainly displaying the embedded JPEG image, which is typically at an earlier position in the file than the actual raw data. So if there's corruption late in the file or the file is incomplete, it will still seem fine in those applications.

iisharankov commented 7 months ago

Hey all. Thanks for the quick replies. I ended up sick with a stomach bug and couldn't so much as leave the bed the last days, apologies for the delayed response.

I hadn't considered that it could be the SD card, which was the same between the MII and MIII. It's a new SanDisk card so it'd be quite surprising, but possible. I agree everything points to a hardware/reading issue now. I'll test with another card these next days and see if there is a difference before closing.

LebedevRI commented 7 months ago

FWIW, this could also be a camera/writing issue.

victoryforce commented 6 months ago

@iisharankov I hope you have successfully found the hardware cause of image data corruption. In any case, this has nothing to do with darktable, so there is no point in keeping this issue open. So, just in case you forget to close the issue after doing your research, I'll close it now.

darktable-org / darktable