Full pipeline is re-run when module near the end if adjusted

[kofa73]

Describe the bug

Caching seems to be ineffective: when I adjust a module param near the end of the pipeline, everything is rerun. In the particular image, several instances of diffuse or sharpen are used, so the processing time can really be felt. It is possible to reproduce this after moving my darktablerc out of the way (so starting with a clean config).

Steps to reproduce

1. Get https://tech.kovacs-telekes.org/files/2024-10-11-darktable-cache/2024-07-14-15-40-14-P1040126.RW2 and https://tech.kovacs-telekes.org/files/2024-10-11-darktable-cache/2024-07-14-15-40-14-P1040126.RW2.xmp
2. Run darktable with -d pipe -d cache
3. Modify e.g. filmic's params (I adjusted the highlights saturation mix). The whole pipeline is rerun.
4. Restore the previous version. The pipeline is run again.

Expected behavior

darktable should not re-evaluate the whole pipeline

Logfile | Screenshot | Screencast

dt.log

Commit

No response

Where did you obtain darktable from?

self compiled

darktable version

4.9.0+794~gba94c624b5

What OS are you using?

Linux

What is the version of your OS?

Ubuntu 24.04

Describe your system?

No response

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

NVIDIA GeForce GTX 1060 6GB; 550.107.02

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[jenshannoschwalm]

Can of course reproduce :-) You may inspect dt_iop_commit_blend_params() and look for return candidate and the relevant caller dt_iop_commit_params() looking for dt_dev_pixelpipe_cache_invalidate_later.

This is quite brute-force but i am still not sure if all situations are met if return candidate would be replaced by return new ? candidate : NULL

Very difficult to test all history scenarios so for now we have to live with this performance penalty.

[jenshannoschwalm]

There seems to be something "even-more-fishy" involved here, investigating...

[kofa73]

I cannot always reproduce this. Could OpenCL settings (async mode) have something to do with it? (Currently, it's disabled, but I may have had it on at one point.)

opencl_synch_cache is described in the manual, but does not appear in my darktablerc - that's an obsolete option, right? My OpenCL and cache-related settings:

cache_color_managed=true
cache_disk_backend=TRUE
cache_disk_backend_full=TRUE
cldevice_v5_nvidiacudanvidiageforcegtx10606gb=0 250 0 16 16 0 0 0 0.000 0.000 0.250
cldevice_v5_nvidiacudanvidiageforcegtx10606gb_building=-cl-fast-relaxed-math
cldevice_v5_nvidiacudanvidiageforcegtx10606gb_id0=600
clplatform_amdacceleratedparallelprocessing=TRUE
clplatform_apple=FALSE
clplatform_intelropenclhdgraphics=TRUE
clplatform_nvidiacuda=TRUE
clplatform_openclon12=FALSE
clplatform_other=FALSE
clplatform_rusticl=FALSE
database_cache_quality=89
opencl=TRUE
opencl_checksum=3380292647
opencl_device_priority=*/!0,*/*/*/!0,*
opencl_library=
opencl_mandatory_timeout=20000
opencl_scheduling_profile=very fast GPU
opencl_tune_headroom=FALSE
performance_configuration_version_completed=17
pixelpipe_synchronization_timeout=200

[ptilopteri]

• Kofa @.***> [10-13-24 13:38]:

  I cannot always reproduce this. Could OpenCL settings (async mode) have something to do with it? opencl_synch_cache is described in the manual, but does not appear in my darktablerc - is that an obsolete option?

does in mine: opencl_synch_cache=active module

-- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc

[kofa73]

• Kofa @.***> [10-13-24 13:38]: opencl_synch_cache is described in the manual, but does not appear in my darktablerc - is that an obsolete option? does in mine: opencl_synch_cache=active module

Probably just a left-over, it's no longer mentioned in the codebase.

[jenshannoschwalm]

@kofa73 yes opencl_synch_cache has been removed some versions ago, it didn't really work and we now have the pipecache holding data.

I couldn't think of anything where OpenCL might be involved or could be related. There will be a pr pretty soon that might help here further as it reports cache invalidation. IIRC in the 4.6 dev cycle there was heavy work on details and raster masks.

Both are very tricky to handle as we would have to possibly recalculate them (if modules before the raster/details "generator" got different params) but the major problem is: we have to make sure that while having a history possibly full of surprises.