search

darktable-org / darktable

darktable is an open source photography workflow application and raw developer
https://www.darktable.org
GNU General Public License v3.0
9.79k stars 1.14k forks source link

-d imageio leads to crash #17112

Closed kofa73 closed 4 months ago

kofa73 commented 4 months ago

Describe the bug

Starting with a clean config, no DB, as soon as I import any images (raw or not, does not matter), darktable crashes with 

*** buffer overflow detected ***: terminated
Aborted (core dumped)

The command to launch was: rm -rf .config/darktable ; rm -rf ~/.cache/darktable/ ; ~/darktable-master/bin/darktable -d imageio

Does not crash with other debug flags, but crashes with -d common (which includes imageio). The crash does not occur under gdb.

Steps to reproduce

Launch with -d imageio. The crash occurs as the lighttable is being populated.

Expected behavior

No response

Logfile | Screenshot | Screencast

No response

Commit

No response

Where did you obtain darktable from?

self compiled

darktable version

4.9.0+57~g3b958b3eff

What OS are you using?

Linux

What is the version of your OS?

Ubuntu 24.04

Describe your system?

Ryzen 5 5600, 64 GB. gcc 13.2.0

I'm attaching my build log for dependency versions.

build.log

Are you using OpenCL GPU in darktable?

None

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

jenshannoschwalm commented 4 months ago

Can't reproduce yet. Any logfile (-d pipe -d imageio) include crash info?

kofa73 commented 4 months ago

No, nothing:

[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     6.9159 [tiff_open] 1326x2000 16bpp, 3 samples per pixel.
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     6.9171 [tiff_open] 1326x2000 16bpp, 3 samples per pixel.
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     6.9183 [tiff_open] 1326x2000 16bpp, 3 samples per pixel.
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     6.9200 [tiff_open] 1326x2000 16bpp, 3 samples per pixel.
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     6.9215 [tiff_open] 1326x2000 16bpp, 3 samples per pixel.
     6.9624 [generate_profile_info] profile `<internal>': color space `RGB '
     6.9629 [generate_profile_info] profile `<internal>': color space `RGB '
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     6.9938 [generate_profile_info] profile `<internal>': color space `RGB '
     6.9959 [generate_profile_info] profile `<internal>': color space `RGB '
     6.9964 [generate_profile_info] profile `<internal>': color space `[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     7.0023 [generate_profile_info] profile `<internal>': color space `RGB '
     7.0026 [generate_profile_info] profile `<internal>': color space `[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18249 (0x4749) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 36867 (0x9003) encountered
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 37398 (0x9216) encountered
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     7.0093 [generate_profile_info] profile `<internal>': color space `RGB '
*** buffer overflow detected ***: terminated
Aborted (core dumped)

4.6.1, built on the same machine, does not crash. 4.8.0 crashes. It did not crash before I upgraded to Ubuntu 24.04, so I assume it's related to a library or GCC change.

I've tried setting GCC to 12 before the build:

export CC=`which gcc-12` ; export CXX=`which gcc-12`

but then I got an error:

-- Trying to query CPU page size
CMake Error at src/external/rawspeed/cmake/Modules/cpu-page-size.cmake:18 (message):
  Failed to query CPU page size:
kofa73 commented 4 months ago

The 4.8.0 AppImage does not crash.

jenshannoschwalm commented 4 months ago

All filetypes? Not just tiff? I tried a lot of "test files" ... no reproducer yet.

kofa73 commented 4 months ago

Raw files (Nikon D7000, Panasonic LX7) also cause a crash, but not JPGs (at least not those exported by darktable).

kofa73 commented 4 months ago

This is my build script, so I assume it's a clean build:

rm -rf ~/.cache/darktable/*kernel*

cd ~/darktable ; 
git checkout master
git clean -d -f -x ;
git pull --rebase
# before updating submodules for the 1st time, disable the integration tests:
# git config submodule.src/tests/integration.update none
git submodule update --init --recursive
rm -rf /home/kofa/darktable-master
./build.sh --prefix /home/kofa/darktable-master --build-type Release --install 2>&1 | tee >/tmp/build.log
kofa73 commented 4 months ago

--build-type RelWithDebInfo also crashes, but not when run under gdb. Any advice on narrowing down compiler switches? --build-type Debug does not crash.

jenshannoschwalm commented 4 months ago

FWIW my script without any problems for long

#!/bin/sh

cd ~/sources/darktable

if [ -d "build" ]; then
  echo "uninstall old builds <sudo>"
  cd build
  sudo make uninstall
  make clean
  cd ..
  sudo rm -fr ~/sources/darktable/build
  sudo rm -fr ~/.cache/ccache
fi

git checkout master
git pull upstream master
git fetch upstream
git submodule update
rm -fr ~/.cache/darktable/cached_v*

./build.sh --prefix /usr/local --disable-kwallet --disable-unity --build-type Release --install --sudo
pehar1 commented 4 months ago

Checked with my test installation of Ubuntu 24.04, fresh install of dt 4.9.0+62~gc18491922c and can confirm.

OS : Linux 6.8.0-38-generic / Ubuntu 24.04 LTS / #38-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun  7 15:25:01 UTC 2024
Memory : MemTotal:        3941248 kB
Xorg : Version: 1:7.7+23ubuntu3
Desktop : LXQt
GTK+ : 3.24.41
gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
CMAKE_BUILD_TYPE : release
OpenCL : no

This is the log of -d pipe -d imageio when trying to import 3 Sony ARW images. Does not crash when importing 3 JPEG images.

~/darktable$ /opt/darktable/bin/darktable --library /home/ph/.config/darktable-fresh/library.db --configdir /home/ph/.config/darktable-fresh/ --cachedir /home/ph/.cache/darktable-fresh/ -d pipe -d imageio
darktable 4.9.0+62~gc18491922c
Copyright (C) 2012-2024 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> ENABLED
  Lua                    -> ENABLED  - API version 9.3.0
  Colord                 -> ENABLED
  gPhoto2                -> ENABLED
  GMIC                   -> ENABLED  - Compressed LUTs are supported
  GraphicsMagick         -> ENABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> ENABLED
  OpenJPEG               -> ENABLED
  OpenEXR                -> ENABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

    35,7230 [xmp_import] creating tag: format|arw
    35,7422 [xmp_import] creating tag: test|sony a77
    36,0364 [avif_open] failed to parse `/home/ph/images/raw/test/DSC00074.ARW': BMFF parsing failed
    36,0366 [rawspeed_open] extensions list to ignore: `cr3'
    36,0385 [avif_open] failed to parse `/home/ph/images/raw/test/DSC00062.ARW': BMFF parsing failed
    36,0385 [rawspeed_open] extensions list to ignore: `cr3'
    36,2248 [avif_open] failed to parse `/home/ph/images/raw/test/DSC00077.ARW': BMFF parsing failed
    36,2249 [rawspeed_open] extensions list to ignore: `cr3'
*** buffer overflow detected ***: terminated
kofa73 commented 4 months ago

@jenshannoschwalm - don't know if this brings up any closer. 588801b23b416efb4e49da785beaef16bc61406e is the first bad commit commit 588801b23b416efb4e49da785beaef16bc61406e Author: hanno@schwalm-bremen.de hanno@schwalm-bremen.de Date: Sun Apr 14 15:38:09 2024 +0200

Code maintenance for imageio export

- some int->gboolean where appropriate
- better readability for enabled modules while exporting and for monochrome detection.

Commenting out all code in imageio.c that prints if DT_DEBUG_IMAGEIO is on did not help. The crash is only avoided if this block is commented out:

    char mbuf[1024] = { 0 };
    for(GList *nodes = pipe.nodes; nodes; nodes = g_list_next(nodes))
    {
      dt_dev_pixelpipe_iop_t *piece = (dt_dev_pixelpipe_iop_t *)nodes->data;
      if(piece->enabled)
      {
        const size_t used = strlen(mbuf);
        snprintf(mbuf + used, sizeof(dev) - used, " %s", piece->module->op);
      }
    }
kofa73 commented 4 months ago

Increasing the buffer to 10K (1024 -> 10024, and at the end of the dtprint 1022 -> 10022 "dt_imageio_export_with_flags") avoids the crash, but I don't see long messages printed:

[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     1.4930 [dt_imageio_export_with_flags]  modules: colorin colorout gamma
     1.4931 [dt_imageio_export] [thumbnail] imgid 15, 1302x656 --> 1302x656 (scale 1.000000). upscale=no, hq=no
     1.5058 [dt_imageio_export_with_flags]  modules: colorin colorout gamma
     1.5058 [dt_imageio_export] [thumbnail] imgid 3, 1326x2000 --> 596x900 (scale 0.450000). upscale=no, hq=no
     1.5289 [dt_imageio_export_with_flags]  modules: colorin colorout gamma
     1.5289 [dt_imageio_export] [thumbnail] imgid 2, 1326x2000 --> 596x900 (scale 0.450000). upscale=no, hq=no
     1.5413 [dt_imageio_export_with_flags]  modules: colorin colorout gamma
     1.5414 [dt_imageio_export] [thumbnail] imgid 5, 1326x2000 --> 596x900 (scale 0.450000). upscale=no, hq=no
     1.5450 [dt_imageio_export_with_flags]  modules: colorin colorout gamma
     1.5450 [dt_imageio_export] [thumbnail] imgid 4, 1326x2000 --> 596x900 (scale 0.450000). upscale=no, hq=no
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
...
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     1.6269 [tiff_open] 1326x2000 16bpp, 3 samples per pixel.
     1.6527 [dt_imageio_export_with_flags]  modules: colorin colorout gamma
     1.6527 [dt_imageio_export] [thumbnail] imgid 11, 1920x1080 --> 1440x810 (scale 0.750000). upscale=no, hq=no
[tiff_open] warning: TIFFReadDirectory: Unknown field with tag 18246 (0x4746) encountered
...
[tiff_open] warning: TIFFFetchNormalTag: ASCII value for tag "Copyright" does not end in null byte
     1.7190 [dt_imageio_export_with_flags]  modules: colorin colorout gamma
     1.7190 [dt_imageio_export] [thumbnail] imgid 6, 1326x2000 --> 596x900 (scale 0.450000). upscale=no, hq=no
     1.7573 [dt_imageio_export_with_flags]  modules: colorin colorout gamma
     1.7573 [dt_imageio_export] [thumbnail] imgid 7, 1326x2000 --> 596x900 (scale 0.450000). upscale=no, hq=no
kofa73 commented 4 months ago

sizeof(dev) is 2816 on my current build, based on 588801.

jenshannoschwalm commented 4 months ago

Yes, some crap there.