Darktable looses all edits on crash

[KristijanZic]

Describe the bug

I’ve noticed that every time Darktable crashes it also looses all the edits made in the latest active editing session. Is there anything that could be done about that behavior? Could we have a ctrl+s shortcut or have Darktable automatically save the changes?

There have been images that I have been working for an hour or two and lost everything because of an ungraceful app shutdown.

To Reproduce

1. Import any image into Darktable and open it in darkroom
2. Make some edit (do not go to another image or exit darkroom)
3. Cause darktable to crash or shut down ungracefully (kill it from terminal or system monitor)
4. Launch Darktable, open the same image and observe that all of your edits are gone.

Expected behavior

Darktable should save changes to disk (xmp file) while editing. It should at least do that periodically and implement a clear label to show when there is unsaved history, similar to DaVinci Resolve. It should also implement CTRL+S shortcut to the save to xmp instead of soft proofing.

Of course, if it can be done so that everything is saved without the need for user awareness and input like CTRL+S, then the CTRL+S shortcut an labels or any GUI is not necessary.

Screenshots

If it can't be done all automatically then here's one possible way to address this:

Platform (please complete the following information):

• Darktable Version: darktable master, darktable 3.4.0
• OS: Ubuntu 20.04

Additional context

[GLLM]

FWIW: I have encountered the same issue w/ dt 3.4 + Ubuntu 20.10

[garrett]

I think everyone who uses darktable for quite some time has lost edits due to a darktable crash at one point or another. (This is true for any pro-level software someone uses a good bit; it's not a knock against darktable.) My work-around is to switch between darkroom and lighttable once in a while. But crashes still sometimes do happen.

I'm guessing darktable doesn't save after every edit as hitting the disk might cause UI lag.

But I wonder if there could be some heuristic where darktable could save in certain cases, such as (just brainstorming here):

• toggling a module on or off (as it usually takes a couple seconds between doing this and adjusting values)
• after applying a preset
• always have a request to save after every module change, but only save under some conditions like:
  • either the mouse hasn't been moved or the keyboard hasn't been used for 5 seconds (or some other value)
  • if the mouse is no longer hovering over the module or the image area (as both are interactive regions where someone)

Or, perhaps, always save the current state as an XMP in /tmp/, which I think is almost always a RAM disk on modern Linux distros and should be quick. Then, if there's a crash on that image, show a recovery dialog (if there's data), then load the XMP and save the database. (Although this isn't a solution for Windows or macOS. But perhaps they can have something similar?

Obviously, it's simpler if there's no freeze in the UI due to disk IO: simply save the current state of the image.

[garrett]

Thanks for making a mockup of an idea, but there are many issues with colors and the history module, as shown in the mockup above:

• many people are colorblind (with different types, with everything between no color to certain colors)
• the UI doesn't explain why the color exists
• constant changes in a UI can be distracting
• red and yellow are often error and warning colors (in many, but not all, cultures)
• not everyone has the history module open at all times

(FWIW: I'm a darktable user, UI designer, software developer, and FOSS fan, but not a darktable developer.)

[johnny-bit]

Personally - I don't like heuristics deciding when and where stuff happens, mostly due to use of many moving parts and timeouts (and I never trust the autosave, games lied to me as a child and left a scar that won't heal). And indication with colour in UI is a bit distracting unless necessary (eg deprecated warning).

The "starters" stuff for saving edits while in darkroom imo would be to have "save" in history module (that's the obvious place IMO) but hidden somewhere in the hamburger menu.

[KristijanZic]

I think that it's a pretty rare case that you loose edits on crash in a software that doesn't have a dedicated save function and is supposed to save automatically. It happens in software that relies on the user to execute a save function himself occasionally. If he doesn't save, he looses edits.

For example, I can't remember the last time I've lost an edit due to crash in Lightroom. I also understand that they don't save history as XMP but as SQLite afaik but I'm just drawing attention to this because the more I'm using darktable the more it happens and I'm beginning to loose edits every day. It's mostly because my RAM gets full when I'm dosing other tasks at the same time and then Darktable crashes resulting in the loss of history. So it's a pretty serious issue.

There are many workarounds: from CTRL-C -> CRTL+V, L -> D to space -> backspace. But that is not a fix and requires too much other processing to take place just to write to XMP. And it's still not automated method it's a workaround to get a user save functionality. Those workarounds are also not documented and obvious so it's an issue that needs some attention.

Now regarding the mockup. It's just an example, by all means do whatever needs to be done to fix it. If it can be done automatically without any UI, great. If you see the titlebar, you'll see I've added the "Edited" text. And that would in my opinion be enough. That's how DaVinci Resolve solved this issue.

* many people are colorblind (with different types, with everything between no color to certain colors)

As for the colorblind people. I'm sorry to ask but how does a colorblind person edit an image exactly? Now if there are some colorblind image editors then color is not an issue for them because they are either trained to recognise it somehow or they use chroma altering glasses like enchroma in which case they should see colors just like non colorblind people do.

* the UI doesn't explain _why_ the color exists

Color yellow is caution and red is a warning. I think that's a pretty universally understood thing. The colors are red and yellow because you are in danger of loosing your edits and your time. It's very important. And it is the most easiest thing to understand just from the usage of the software. And it would be like this; The first time user sees it, he wonders what it is. Then he executes CTRL+S and notices the colors are gone. And he immediately ties those warnings with him having to execute save. It's probably the only thing in darktable that user will actually understand from the usage in a few minutes. And it should also be explained more in the documentation for those that need a little more explanation or are curious to know for sure sure.

* red and yellow are often error and warning colors (in many, but not all, cultures)

Well, I'm sorry for the other cultures then. But actually I'm not because you're wrong here. This IS universal everywhere. Everywhere where they drive cars at least. Green is safe, yellow/orange is caution and red is warning. I'm sorry if I'm wrong but even then that's probably not the targeted audience. If they don't have at least one traffic light in the country then probably they don't have computers or electricity let alone money to spend on cameras.

* changes in a UI can be distracting

As for it being distracting, it's designed to draw attention. You are risking loosing your data that you've spend the last 3 hours or so. It better be distracting. It's an urgent warning.

* not everyone has the history module open at all times

As for not having history module opened all the time, you have the gray, yellow, red "Edited" text in the titlebar to suplement it. And probably just that would be more than enough without any colors in history. This was just a fast example I came up with. But by all means, let's imagine a better solution.

But in the end, I'd be much happier if I didn't have to think about saving at all.

[parafin]

I agree that save action should be somehow triggerable without switching views/images. Not sure any automatic heuristic is needed though.

[parafin]

As for crashes - really the source of crashes should be fixed, not workarounds created for it. If you have too little RAM - then add RAM or/and swap space.

[TurboGit]

Not sure any automatic heuristic is needed though.

But it will always crash when you forgot to save :) That's the LEM in French or Murphy's Law.

I would be more in favor of saving the XMP every x seconds (or x modifications) for example. That is, when the history change we record a status and when needed we do trigger a save xmp or a database history write or both. No idea at this stage.

[KristijanZic]

As for crashes - really the source of crashes should be fixed, not workarounds created for it. If you have too little RAM - then add RAM or/and swap space.

One can never have enough RAM in this age and I have 8GB on my laptop and 64 on desktop. That's not an issue. The issue is; Darktable has no autosave or even save functionality. That means that the user is always at risk of loosing my current work no matter the cause.

[KristijanZic]

I would be more in favor of saving the XMP every x seconds (or x modifications) for example. That is, when the history change we record a status and when needed we do trigger a save xmp or a database history write or both. No idea at this stage.

Me too, but I'd still like to have that "Edited" indicator I've shown in the mockup and a CTRL+S shortcut. But if I had to pick one or another, I'd pick what you said; having Darktable just autosave everything for me.

[TurboGit]

No please, no more color in the UI.

[TurboGit]

I think everyone who uses darktable for quite some time has lost edits due to a darktable crash at one point or another.

I had one such case in 7 years of use and developing more than 20.000 pictures.

[KristijanZic]

I think everyone who uses darktable for quite some time has lost edits due to a darktable crash at one point or another.

I had one such case in 7 years of use and developing more than 20.000 pictures.

It doesn't matter. If you there is a power outage in your block and you're editing, you'll still loose it again. The issue is the software. Users shouldn't be required to get the UPS, get more RAM, do a dedicated Darktable machine etc to minimize loosing data because Darktable doesn't have a save shortcut. That's like Gimp offering a save function only when exiting a project or changing the project tab.

[garrett]

As for the colorblind people. I'm sorry to ask but how does a colorblind person edit an image exactly?

I know colorblind photographers (and designers and illustrators too). When a colorblind person is editing an image, they either rely on the program to have accurate colors (and colorblind people who do see color can still get an idea, including saturation) or they intentionally output to black and white photos.

As for design and illustration, they often have palettes and color relationships in their heads as a theoretical thing and constantly think about it all the time. Then they get another designer who has full color vision to double-check their work. I think it is often the same for photography too, except the software (in this case, darktable) should give them accurate-ish colors by default.

---

@KristijanZic: If you're running out of RAM and are on Linux, perhaps consider using zRAM for swap. Fedora 33 enabled this by default (and Fedora 34 will probably adjust it a little). It's not too hard to enable it for other distros too.

My personal experience with zRAM swap: Even though I have a ton of RAM (16 GB personal laptop, 20 GB work laptop), I do hit swap from time to time. zRAM is almost penalty-free as the tiny CPU overhead is much better than the disk IO overhead, and zRAM is only used when it is needed. It's almost like free RAM. Additionally, on my Raspberry Pi home servers, zRAM swap makes sure it doesn't hit the SD card (which has very limited rewrites and can easily stop working) and speeds up the little devices.

FWIW: Windows 10 now does something similar by default and macOS from Mavericks and above does too. Android also has been using compressed RAM for a few years as well, where it's really needed most (as phones have often only have RAM in the lower single digits of GB).

[TurboGit]

f you there is a power outage in your block and you're editing, you'll still loose it again.

No, I'm on a laptop :) But that's not the issue :) And BTW I have also a small UPS so I won't loose my Internet connection nor my home server.

[garrett]

I had one such case in 7 years of use and developing more than 20.000 pictures.

That's excellent.

However, my own anecdotal experience: I was working on around 160 photos for printing at the end of last year and had a handful of crashes. Sometimes I was working on an image for half an hour or more (often having to retouch things like dust spots or cigarette butts out of an image) and lost all that work. Some images and/or modules might be a little more buggy than others.

While I agree that darktable should not crash, it also should be a little more aggressive about saving in the darkroom.

Having a shortcut or button to manually save doesn't fix the problem. It becomes just another thing to needlessly hit.

Is it possible to trap exceptions, write out an XMP of in-memory state, and then have a way to recover it on next launch? That would handle crashes, but wouldn't address power outages (which are hopefully much less common than the already somewhat uncommon crash).

[KristijanZic]

f you there is a power outage in your block and you're editing, you'll still loose it again.

No, I'm on a laptop :) But that's not the issue :) And BTW I have also a small UPS so I won't loose my Internet connection nor my home server.

Well, you're rich. I'm not. I think there was a survey that showed that Darktable is used mostly by highly educated people from rich parts of the world. This would explain the lack of understanding for features like this that could be mitigated to a degree by the users themselves if they have the money for their own home UPS and servers.

I wrote this harshly on purpose to make a quick point (I know you're probably not actually very rich so don't take this to heart). But that said, I think that's very cool. I'd love to do that at home too one day :D

The issue is the software. Users shouldn't be required to get the UPS, get more RAM, do a dedicated Darktable machine etc to minimize loosing data because Darktable doesn't have a save shortcut. That's like Gimp offering a save function only when exiting a project or changing the project tab.

[TurboGit]

This would explain the lack of understanding for features like this

I've not read here that people where against such feature.

[KristijanZic]

If you're running out of RAM and are on Linux, perhaps consider using zRAM for swap. Fedora 33 enabled this by default (and Fedora 34 will probably adjust it a little). It's not too hard to enable it for other distros too.

I'm not I just have 2k+ firefox tabs open (I'm not even kidding, I literally do). But this isn't the issue. I mas be living in an area with constant power outages and without money to invest in a UPS or a laptop.

On the colorblind issue. I know of designers too, they used color palets but nowdays just wear enchroma glasses. I know of photographers too, they shoot great shots. But I really don't know of any photo editors. They can't rely on palettes with Darktable. Maybe on a color picker? I'd really love to find a colorblind photo editor to explain their process. It must be fascinating. They probably use enchroma glasses. I can't imagine any other way.

[KristijanZic]

I've not read here that people where against such feature.

Oh, my bad. I got the idea that people were telling/mocking me to buy more RAM, use drive cash and buy a UPS to fix the issue instead of focusing on the software. Totally my bad, I completely misinterpreted. I'm so so so sorry for that. It's a language barrier and sleep depravation :P Thanks for all the advice!

@garrett Yeah, I have a large cash file on my desktop, a dedicated ssd for that actually. But I have just 500gb ssd on my laptop so I don't want to ruin my only ssd for cashe. It does ruin your SSD much quicker. And for what? For FireFox? :D

[garrett]

\<aside>

I'd really love to find a colorblind photo editor to explain their process. It must be fascinating. They probably use enchroma glasses. I can't imagine any other way.

Photography was black and white before color was introduced. A lot of people still shoot black and white as their own preference (and some darktable users do this too), even if they're not colorblind. And darktable has a lot of ways to convert raw photos to black and white (with different pros and cons depending on the method).

But, specifically: One of my friends is a colorblind designer who enjoys photography and most of his photos are actually in color. He's red-green colorblind (if I recall correctly). He still looks for framing, tonality, and saturation just like us full-spectrum color sighted folks and can usually determine if something is color neutral. And his camera and photograph editing apps all work hard to give him a good starting point (just like they do for us). Plus, there are indicators if something is oversaturated, and so on.

For someone fully colorblind (which is quite rare, but does happen), they can still rely on the software to get it about right. Or prefer black and white images.

---

Also, my suggestion about zRAM was based on personal experience at how it works much better. I didn't mean it as a workaround for this issue. And zRAM actually prevents your computer from hitting the SSD for swap. It hits the compressed dynamic swap in RAM first and then hits your swap file or partition (if you even have one), saving the write cycles of your device. (Which, for modern SSD, is actually much longer than people fear. But still, they do have limited lifespans.)

My swap is used for a lot of things. I'm doing software development, running Inkscape and GIMP, having multiple browsers open (some with a "lot" of tabs — nothing like your thousands of tabs, though), and a ton of different Electron-based (*sigh*) windows open... and all of this at the same time, on 3 screens. It does eat RAM sometimes.

\</aside>

---

But back to the actual issue: darktable still needs to be able to save in-progress editing to prevent data loss for crashes, power outages, OS crashes, shutting the computer off while darktable is running, etc.... somehow. And it needs to be able to do so without impacting the UI responsiveness.

(I don't think people are debating this?)

It's generally stable, but when some big interruption happens during editing, it's not great at the moment when work, time, and effort are lost.

[johnny-bit]

let's go back to the core issue, no need to discuss who has what ram/ups/os combo.

I still propose: a "save" menuitem in history module. from that a heuristic can be made for determining when to save, like simple timeout timer that'll trigger save every x seconds when there is nothing in progress (actually saving to xmp/db is very fast because the save is super quick). An indication for save state is up to designers... One idea could be a :black_circle: next to history module name similarly to "changed prefs" in preferences could say that the changes aren't saved and just disappear once save is completed?

As for crashes: let's make sure there are less and less crashes possible :P

[aurelienpierre]

I had one such case in 7 years of use and developing more than 20.000 pictures.

Crashes with loss of data happen to me every week, and indeed it's mightily annoying. The only work-around I have found is to compress history between every module setting. Also, some bugs need to be tested in "real-life" conditions, in production, because they happen only on heavy edits and such, so having a safety jacket to retain at least the history would be great.

[parafin]

Don't forget that just because application has written some data to file/DB, doesn't mean that crash or power interruption won't destroy it. You have to flush the file to the disk to be sure, which isn't free operation and affects other I/O on the system. Keep that in mind when designing the implementation of this feature. Also I'm hoping we're not considering frequent crashes as the use-case to optimize for. That being said, trying to save the state is of course a good idea and nobody is against it. We just have to make sure it won't lock (consider network FS or spinning disk on laptop that entered sleep state) any other threads, processing or GUI, which might be not as easy as it sounds.

[github-actions[bot]]

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[daniel-j-h]

Commenting here since https://github.com/darktable-org/darktable/issues/8768 got closed as duplicate :two_men_holding_hands: and the stale bot seems to :hocho: this ticket :see_no_evil:

Summarizing the ticket, so far I have seen arguments

• We should fix the underlying crashes instead of a save mechanism
• Anecdotally "for me" crashes have happen rarely
• Automatically saving based on heuristics can be unintuitive for the user
• Saving to disk is expensive in terms of compute and IO

As an engineer myself - I understand :bow:

But putting on my user hat :tophat:

• The longer I work on an image the higher the chance that something will crash
• The longer I work on an image, the higher the frustration if a crash happens for whatever reason

At the same time there is already a save feature implemented it's just not called this way: switching to the lighttable and back. This saves the xmp sidecar files and nothing is stopping users from manually switching the view every half an hour or so to "save".

What would be the simplest iteration from this lighttable view switch to a basic save mechanism? Can we put the view switching xmp save logic behind a simple save button, or on ctrl+s, or somewhere?

It's not about handling the theoretical cosmic bit flips :night_with_stars: it's about not discouraging the user from ever coming back due to losing half a day of work. Falling down is fine, standing up is what counts :adhesive_bandage:

[ptilopteri]

have you tried committing a key combination (short cup) to (gtk_accel_path "/modules/copy_history/write sidecar files" "")

look in ~/.config/darktable/keyboardrc

[github-actions[bot]]

This issue did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[garrett]

Still an issue with darktable, not stale.

[github-actions[bot]]

This issue did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[garrett]

Still an issue when crashes happen.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[Dakes]

Still an issue. Happened to me multiple times in the last few days alone.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[KristijanZic]

Still an issue.

[jenshannoschwalm]

I would be more in favor of saving the XMP every x seconds (or x modifications) for example. That is, when the history change we record a status and when needed we do trigger a save xmp or a database history write or both. No idea at this stage.

@TurboGit I started working on this, the writing of the sidecar is not the problem for me but how would i make sure the database file is written at some time so it would have all things done dt is started again?

[TurboGit]

@jenshannoschwalm : As a summary I'm not sure we can do something good for dt here.

The SQLite DB has atomic read and hot journal, so using it should be safe across crashes and be able to recover data.

My concern is about speed. Write to the DB & XMP commit params each time an action is done on the GUI will ruin all the speed up (and more) that we have gain recently. Is that a good move? Disk are so slow and even slower if one work on a network drive.

All in all I still prefer loosing last edit than having a slow UI. Of course one will say let's just commit params every 10s or 30s... Still we will loose some data in between, is that really better?

Of course you may prove me wrong and come with some better idea and fast implementation :) That's just my current feeling.

[jenshannoschwalm]

Dear pascal, no chance for performance drop from my side too :-) BTW the only crashes I had in the last months was while working on astrodenoise and raster masks...

Will do a proposal pr and let's discuss that.

[crepererum]

I think this somewhat performance thing kinda depends on the expectations here: if you really want to survive ANY kind of crash (including your PC/OS) you MUST call fsync. This will be very slow and to my knowledge, no editor does that for EVERY edit / slider change. However you could do that in regular intervals (like every 30s) and most people would still get good value out of it, because there's a difference between loosing 30s of work vs loosing an hour of work. So to answer

All in all I still prefer loosing last edit than having a slow UI. Of course one will say let's just commit params every 10s or 30s... Still we will loose some data in between, is that really better?

I think yes, it's better.

[Jiyone]

All in all I still prefer loosing last edit than having a slow UI. My concern is about speed.

Define slow UI please because I thought darktable fast enough now, so taking the time to save your work at intervals wouldn't be an issue on that side? Here, it's only taking 0.11 seconds to save, which is quite fast.

This issue happens sometimes on my side too and it annoys me. Since when the speed is more important than people loosing work?

Still we will loose some data in between, is that really better?

Why asking ? It will be 30s max of lost work anyway, which is better than loosing 1h, isn't it? Hmm?

[ptilopteri]

or maybe a keystroke option to write an xmp which would contain the edit.

[Jiyone]

or maybe a keystroke option to write an xmp which would contain the edit.

You can shortcut the button Compress history stack and edit the setting write sidecar file for each image to after edit. This will update the XMP file.

[ptilopteri]

or maybe a keystroke option to write an xmp which would contain the edit.

You can shortcut the button Compress history stack and edit the setting write sidecar file for each image to after edit. This will update the XMP file.

which would provide a recovery point using import

[jenshannoschwalm]

Some noise here :-) Don't you recognize that we will check if there is something good to be done for this issue?

[wpferguson]

Some thoughts...

I wouldn't use a timed write. If darktable crashed in the middle of a timed write, a corrupt database could be the result.

I would do writes after a history stack change, and then just update the database (since that's the primary storage for the edits). Write the xmp files on image change, since they are a backup for the database.

Updating after a history stack change also protects the database if the crash is iop related. If the iop causes the crash, then the history stack change event will never trigger and a database write wont be in progress at the time of the crash..

[crepererum]

I wouldn't use a timed write. If darktable crashed in the middle of a timed write, a corrupt database could be the result.

I agree with the rest of your statement but wanna point out that this bit is misleading. With both write strategies (timed or history stack based) you should use file change mechanism that doesn't corrupt if it's interrupted. If the state is kept in sqlite, you get this for free. If you overwrite the sidecar file, then you should write to a temporary file at the same directory first and then use the OS "move" operation to do the actual override (to my knowledge this OS operation is atomic on all systems, especially because this is on the same file system). This way you don't end up with half written / corrupted sidecar files.

[jenshannoschwalm]

Should be solved so closing this for now