Wavelet Denoise crash Rawtherapee when preview size is 1%

[syyrmb]

Short description When zooming out the preview to 1%, then enable Wavelet tool and its Denoise and Refine subtool, then apply denoise slider (the order to do the 3 things should be unimportant), Rawtherapee crashes.

Steps to reproduce

1. Open a Raw file
2. Enable Wavelet tool
3. Enable its Denoise and Refine tool
4. Adjust denoise slide (I only tested level 1's denoise slider)
5. Zoom out the preview to 1%

Expected behavior Rawtherapee should keep running.

Additional information

• Version of RawTherapee: Latest Release 5.10 (Also replicated in the debug build 8adb5439d)
• Version of your operating system: Windows 10

[Benitoite]

I was able to reproduce this crash with dev on x86_64 macOS 12.5.7 Monterey

from the crash report:

Crashed Thread:        36

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000028
Exception Codes:       0x0000000000000001, 0x0000000000000028
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [47054]

Thread 36 Crashed:
0   rawtherapee                            0x10457c904 rtengine::ImProcFunctions::WaveletDenoiseAllL(rtengine::wavelet_decomposition&, float*, float (*) [3], float*, int, int) + 324
1   rawtherapee                            0x10479c8f2 .omp_outlined..18351 + 8994
2   libomp.dylib                           0x105088d83 __kmp_invoke_microtask + 147
3   libomp.dylib                           0x105037c1a __kmp_serial_fork_call(ident*, int, fork_context_e, int, void (*)(int*, int*, ...), int (*)(int), kmp_info*, kmp_team*, ompt_data_t*, void**, ompt_data_t**, __va_list_tag*) + 945
4   libomp.dylib                           0x105036bf5 __kmp_fork_call + 3928
5   libomp.dylib                           0x10502b6d9 __kmpc_fork_call + 283
6   rawtherapee                            0x104799772 rtengine::ImProcFunctions::ip_wavelet(rtengine::LabImage*, rtengine::LabImage*, int, rtengine::procparams::WaveletParams const&, rtengine::WavCurve const&, rtengine::WavCurve const&, rtengine::WavCurve&, rtengine::Wavblcurve const&, rtengine::WavOpacityCurveRG const&, rtengine::WavOpacityCurveSH const&, rtengine::WavOpacityCurveBY const&, rtengine::WavOpacityCurveW const&, rtengine::WavOpacityCurveWL const&, LUT<float> const&, int) + 9938
7   rawtherapee                            0x10452a7ad rtengine::Crop::update(int) + 37533
8   rawtherapee                            0x104521094 rtengine::Crop::fullUpdate() + 116
9   libglibmm-2.4.1.dylib                  0x1051703c9 (anonymous namespace)::call_thread_entry_slot(void*) + 35
10  libglib-2.0.0.dylib                    0x105a353e5 g_thread_proxy + 66
11  libsystem_pthread.dylib             0x7ff80cb024e1 _pthread_start + 125
12  libsystem_pthread.dylib             0x7ff80cafdf6b thread_start + 15

[Lawrence37]

This is probably due to the maximum wavelet level getting reduced for smaller images (like a zoomed-out preview) and the code tries to use higher wavelet levels that do not exist. There needs to be a validation on the wavelet level in class wavelet_decomposition.

[Desmis]

I open a Pull-Request #7169

In the ipwavelet.cc code, the denoise case is special because it requires a fixed number of decomposition level allocations (7).

This assumes, for this particular case, a window of at least 128 pixels.

I modified the code so that this minimum value is assigned if a window is larger than 128. In other words, if the window is too small, denoise is disabled.

[Benitoite]

Looks good @Desmis , the fix works for me- can't get it to crash.

[Lawrence37]

Preliminary fix merged to branch release-5.11.