segfault when opening DNG file

[stefantalpalaru]

Using the "dev" branch HEAD:

Thread 165 "rawtherapee" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffeb10d640 (LWP 14734)]
0x0000555555f3a0ae in LUT<float>::operator[]<float, float, void> (this=<optimized out>, index=-nan(0x400000)) at /src/77_DLD/CODE/00_github/RawTherapee/rtengine/LUT.h:481
481         T p1 = data[idx];
(gdb) bt
#0  0x0000555555f3a0ae in LUT<float>::operator[]<float, float, void>(float) const (this=<optimized out>, index=-nan(0x400000))
    at /src/77_DLD/CODE/00_github/RawTherapee/rtengine/LUT.h:481
#1  _ZN8rtengine15ImProcFunctions7rgbProcEPNS_10ImagefloatEPNS_8LabImageEPNS_13PipetteBufferERK3LUTIfESA_SA_iSA_SA_SA_ffRKNS_18ColorGradientCurveERKNS_12OpacityCurveEbSA_SA_RKNS_9ToneCurveESJ_SJ_SJ_RdSK_SK_RfSL_SL_diiPNS_10DCPProfileERKNS_20DCPProfileApplyStateERS7_IjEmb._omp_fn.0(void) ()
    at /src/77_DLD/CODE/00_github/RawTherapee/rtengine/improcfun.cc:2391
#2  0x00007ffff59aa156 in  () at /usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libgomp.so.1
#3  0x00007ffff55142ba in start_thread (arg=<optimized out>) at pthread_create.c:442
#4  0x00007ffff559653c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) demangle _ZN8rtengine15ImProcFunctions7rgbProcEPNS_10ImagefloatEPNS_8LabImageEPNS_13PipetteBufferERK3LUTIfESA_SA_iSA_SA_SA_ffRKNS_18ColorGradientCurveERKNS_12OpacityCurveEbSA_SA_RKNS_9ToneCurveESJ_SJ_SJ_RdSK_SK_RfSL_SL_diiPNS_10DCPProfileERKNS_20DCPProfileApplyStateERS7_IjEmb._omp_fn.0
rtengine::ImProcFunctions::rgbProc(rtengine::Imagefloat*, rtengine::LabImage*, rtengine::PipetteBuffer*, LUT<float> const&, LUT<float> const&, LUT<float> const&, int, LUT<float> const&, LUT<float> const&, LUT<float> const&, float, float, rtengine::ColorGradientCurve const&, rtengine::OpacityCurve const&, bool, LUT<float> const&, LUT<float> const&, rtengine::ToneCurve const&, rtengine::ToneCurve const&, rtengine::ToneCurve const&, rtengine::ToneCurve const&, double&, double&, double&, float&, float&, float&, double, int, int, rtengine::DCPProfile*, rtengine::DCPProfileApplyState const&, LUT<unsigned int>&, unsigned long, bool) [clone ._omp_fn.0]

$ cat AboutThisBuild.txt 
Version: 5.8-3098-g404a584c7
Branch: dev
Commit: 404a584c7
Commit date: 2022-06-12
Compiler: cc 11.3.0
Processor: AMD\ FX-8320E\ Eight-Core\ Processor
System: Linux
Bit depth: 64 bits
Gtkmm: V3.24.6
Lensfun: V0.3.3.0
Build type: Debug
Build flags: -O3 -march=native -mlwp -pipe -ggdb -std=c++11 -fno-tree-loop-vectorize  -Werror=unused-label -Werror=delete-incomplete -fno-math-errno -Wno-attributes -Wall -Wuninitialized -Wcast-qual -Wno-deprecated-declarations -Wno-unused-result -Wunused-macros -fopenmp -Werror=unknown-pragmas -g -ftree-vectorize
Link flags: -Wl,-O1 -Wl,--as-needed -Wl,--sort-common
OpenMP support: ON
MMAP support: ON
Build OS: Linux 5.17.9-gentoo x86_64
Build date: Wed, 15 Jun 2022 20:30:06 +0000 UTC
Build epoch: 1655325006
Build UUID: 69629982-96f1-4df1-8aac-787bfa3af397

The segfault happens even when I disable OpenMP and TCMalloc.

I'm trying to open a DNG file produced by DJI Osmo Action, but the first DNG sample on this page also triggers the segfault: https://www.kenrockwell.com/leica/m9/sample-photos-3.htm

[heckflosse]

but the first DNG sample on this page also triggers the segfault

works fine with this:

$ cat Release/AboutThisBuild.txt
Version: 5.8-3098-g404a584c7
Branch: dev
Commit: 404a584c7
Commit date: 2022-06-12
Compiler: gcc 12.1.0
Processor: undefined
System: Windows
Bit depth: 64 bits
Gtkmm: V3.24.5
Lensfun: V0.3.2.0
Build type: Release
Build flags: -Wno-parentheses -std=c++11 -fno-tree-loop-vectorize -march=native -Werror=unused-label -Werror=delete-incomplete -fno-math-errno -Wno-attributes -Wall -Wuninitialized -Wcast-qual -Wno-deprecated-declarations -Wno-unused-result -Wunused-macros -fopenmp -Werror=unknown-pragmas -O3 -DNDEBUG -ftree-vectorize
Link flags:  -march=native
OpenMP support: ON
MMAP support: ON
Build OS: MINGW64_NT-10.0-19043 3.3.5-341.x86_64 x86_64
Build date: Thu, 16 Jun 2022 10:19:58 +0000 UTC
Build epoch: 1655374798
Build UUID: 310a7c7c-f0fa-4192-bcbe-5ce8c94c492a

[stefantalpalaru]

I found a way to replicate it on CPU families other than Piledriver:

cd build

rm -f CMakeCache.txt; CFLAGS="-O3 -mfma -mtune=bdver2 -ggdb" CXXFLAGS="$CFLAGS" LDFLAGS="" cmake -DENABLE_TCMALLOC=ON -DOPTION_OMP=ON -DCMAKE_BUILD_TYPE="release" -DPROC_TARGET_NUMBER=0 -DBUILD_BUNDLE=ON -DBUNDLE_BASE_INSTALL_DIR="$HOME/programs/rawtherapee" ..

make clean; make -j8 install

gdb ~/programs/rawtherapee/rawtherapee

[stefantalpalaru]

I can no longer replicate the problem in the latest Git HEAD.