Corrupted image if copy history stack contains 2 exposures.

vancouverbluesea commented 3 years ago

Describe the bug/issue I have tried to copy a stack that happen to have 2 exposures. The second exposure is with gradient mask. Once the stack was copied - it did not take the second exposure - it transferred in the history but not as a module.

To Reproduce Please provide detailed steps to reproduce the behaviour, for example:

Process an image with 2 exposures
Copy the history stack to the new image (the complete history stack)

Expected behavior The complete history should be transferred

Screenshots (if applicable)

Screencast (if applicable) Movie is below https://www.dropbox.com/s/8lurfar7mlaqhz5/2021-03-24%2017-45-43.mkv?dl=0

Platform Please fill as much information as possible in the list given below. Please state "unknown" where you do not know the answer and remove any sections that are not applicable

darktable version : e.g. 3.4.1 OS: Pop!_OS 20.04 LTS x86_64 Host: Serval WS serw12 Kernel: 5.8.0-7642-generic Uptime: 1 day, 1 hour, 30 mins Packages: 2505 (dpkg), 57 (flatpak) Shell: bash 5.0.17 Resolution: 2560x1440, 1920x1080 DE: GNOME WM: Mutter WM Theme: Pop Theme: Pop [GTK2/3] Icons: Pop [GTK2/3] Terminal: gnome-terminal CPU: AMD Ryzen 9 3900 (24) @ 3.100GHz GPU: NVIDIA 07:00.0 NVIDIA Corporation Device 1f14 Memory: 7569MiB / 64275MiB

Additional context attaching the xmp used 20210126_122146_4568.CR2.zip

vancouverbluesea commented 3 years ago

https://github.com/darktable-org/darktable/issues/8330#issue-816644971 very similar to the above issue

vancouverbluesea commented 3 years ago

https://github.com/darktable-org/darktable/issues/7655#issue-777643103 https://github.com/darktable-org/darktable/issues/8049#issue-797762378 Reading some of the previous issues - looks like the expectation is that it maybe corrupted XMP. However - I can reproduce it over and over and it is a brand new image, brand new processing. I don't think it is a corrupted xmp (that originates the issue).

TurboGit commented 3 years ago

Probably already fixed.

Nilvus commented 3 years ago

#7655 (comment) #8049 (comment) Reading some of the previous issues - looks like the expectation is that it maybe corrupted XMP. However - I can reproduce it over and over and it is a brand new image, brand new processing. I don't think it is a corrupted xmp (that originates the issue).

Or a corrupted database lines from original image (or target one) where those 2 instances of same modules were created. This is an issue fixed in a previous version of darktable. The issue was a wrong writing when 2 instances are used for same module, but it's not possible to fix already made bad writings. So nothing to do probably in darktable source code.

You could fix it by duplicating images affected (AND source image where you copy history stack), then copy history on duplicate one (use original duplicate, not same one) and you will have good history. As explains in closed issues you pointed.

I just tried with your XMP in one of my image. If I apply your XMP, I have a corrupted image. I then duplicate with original button (and not clone one) in darkroom duplicate manager, to have an empty history. Then I copy the history from the corrupted image to the original clone just created, and history is correctly applied. You then just have to completely delete corrupted one. Annoying but the original issue is fixed and will not reproduce on future images.

TurboGit commented 3 years ago

Can't this be #7477 which is already fixed?

vancouverbluesea commented 3 years ago

#7655 (comment) #8049 (comment) Reading some of the previous issues - looks like the expectation is that it maybe corrupted XMP. However - I can reproduce it over and over and it is a brand new image, brand new processing. I don't think it is a corrupted xmp (that originates the issue).

Or a corrupted database lines from original image (or target one) where those 2 instances of same modules were created. This is an issue fixed in a previous version of darktable. The issue was a wrong writing when 2 instances are used for same module, but it's not possible to fix already made bad writings. So nothing to do probably in darktable source code.

You could fix it by duplicating images affected (AND source image where you copy history stack), then copy history on duplicate one (use original duplicate, not same one) and you will have good history. As explains in closed issues you pointed.

I just tried with your XMP in one of my image. If I apply your XMP, I have a corrupted image. I then duplicate with original button (and not clone one) in darkroom duplicate manager, to have an empty history. Then I copy the history from the corrupted image to the original clone just created, and history is correctly applied. You then just have to completely delete corrupted one. Annoying but the original issue is fixed and will not reproduce on future images.

I really had to read the explanation several times. But yes - I admit it works. Thank you!

vancouverbluesea commented 3 years ago

Can't this be #7477 which is already fixed?

I think this issue describes it well indeed. I am going to move on with the workaround as @Nilvus pointed out. Thank you for all your help!

vancouverbluesea commented 3 years ago

@Nilvus I really believe this issue is not fixed in DT. If we are to assume that something was wrong either in the original or target image or in the DB - then starting a fresh - fresh DB, fresh source, fresh target - it should be good. But it is not. I do use the workaround (with copy original) and I am very grateful for it. But would you be able to either test once again with fresh DB, brand new images (never imported) - or if I am completely misunderstanding - would you be so kind to point me where does my logic needs correction?

TurboGit commented 3 years ago

I really believe this issue is not fixed in DT. If we are to assume that something was wrong either in the original or target image or in the DB - then starting a fresh - fresh DB, fresh source, fresh target - it should be good. But it is not.

In which version it is not fixed ? What did you tried ?

vancouverbluesea commented 3 years ago

@TurboGit I have DT 3.4.1 and I am on PopOS 20.04

I tried to delete (rename) ~/.config/darktable When I started DT it created the folder so it was on a fresh DB - no images at all. I didn't change any configuration either. Then I imported 2-3 images (that are new - no xmp) and only changed exposure on one of the images - 2 instances of exposure. Moved back to light table and selected the complete stack by ctrl+shift+c and made sure all is selected then ctrl+v on the new image. The result is a broke image (the one that is a target)

So - if the database is fresh, the images are new - no .xmp to start with - how do I end up with a broken image? I understood that https://github.com/darktable-org/darktable/issues/7477 is already fixed - so 3.4.1 has a fix. Did I misunderstood?

Reading the above from @Nilvus https://github.com/darktable-org/darktable/issues/8547#issuecomment-807267360 - it makes complete sense that the issue is in one of 3 places - original image, target image or DB and each one of these can introduce a corruption. So - I am very thankful for the workaround provided. However - my understanding is that this is a workaround - not the final solution. Was my understanding wrong?

Nilvus commented 3 years ago

What you describe is quite strange. Maybe there's something remaining with some specific settings. I rarely use 2 exposures instances but I never see this issue since a time, except rarely on my side with old images. So ones with XMP and/or database where history is corrupted. And if the workaround works, that means that it's a writing error that the workaround solved. It would be good to share a XMP/RAW with the issue (a fresh one of course!).

@TurboGit: if you see something else here...

vancouverbluesea commented 3 years ago

@Nilvus

Here are the files and description. I am providing step by step the process with both the DB (the complete dartkable folder) and the .XMP - please use what you need.

xmp_demo of corruption.zip xmp_initial import.zip xmp_workaround complete.zip xmp_workaround_dup manager copy original.zip darktable_demo of corruption.zip darktable_import2images.zip darktable_initial.zip darktable_workaround complete.zip darktable_workaround_dup manager_copyoriginal.zip RAW files are here https://www.dropbox.com/s/8sluenajbh4fis4/RAW%20files.zip?dl=0

The steps were as follow

renamed ~/.config/darktable
New dartkable folder as created by DT - no config done. "darktable_initial.zip"
Imported 2 images - no initial .xmp all is brand new "darktable_import2images.zip" "xmp_initial import.zip"
Demo of corruption - "20210417_132708_5019.CR2" has 2 exposures (just created). Copy "all" to "20210417_132709_5020.CR2" and corruption is visible. "darktable_demo of corruption.zip" "xmp_demo of corruption.zip"
Starting workaround by creating a copy original with duplication manager "darktable_workaround_dup manager_copyoriginal.zip" "xmp_workaround_dup manager copy original.zip" - this results in new XMP - "20210417_132709_5020_01.CR2.xmp"
Complete the workaround by copying all from "20210417_132708_5019.CR2.xmp" to "20210417_132709_5020_01.CR2.xmp" - the workaround works as expected - correct copy of the stack. "darktable_workaround complete.zip" "xmp_workaround complete.zip"

TurboGit commented 3 years ago

I understood that #7477 is already fixed - so 3.4.1 has a fix. Did I misunderstood?

No this ifx is not in 3.4.1 and will only be in upcoming 3.6. So I'll close this and we will revisit if you can still reproduce on 3.6 when release.

darktable-org / darktable

Corrupted image if copy history stack contains 2 exposures. #8547