Improve scratch layer "Save" UI

[jcpayne]

Feature description. Improve the Save dialogues for scratch layers so that users are less likely to lose their work by mistake. Also, warn when a process creates a scratch layer by default.

Additional context I just lost a whole day of someone else's work by closing a QGIS project and ignoring the warning that two scratch layers hadn't been saved. I am an experienced QGIS user but I didn't know what a scratch layer was, and I had "saved" the layers multiple times--or so I thought.

A) I created two scratch layer unintentionally at some point during the process of renaming two existing layers and adding auto-incrementing variables using the Processing toolbox. I would have opted for creating a regular layer instead of a vanishing temporary scratch layer if I had realized what was happening. B) I had "saved" the scratch layers at least 50 times, using the Save buttons on the toolbar and the "Save edits" button in the Attribute table. At no time did a dialogue give any indication that I was going to lose the layer anyway when I closed the project. I had also saved the Project multiple times. This seems like a clear example of bad UI design to me. "Save" has a specific meaning to users that transcends any particular piece of software, and QGIS's "Save" behavior for scratch layers violates that meaning. C) When I closed the project, I got a warning that a scratch layer existed, but no information on what a scratch layer was or how to deal with it. I took a chance that it didn't matter, and was wrong. A hyperlink or a more informative dialogue would have helped.

It would be easy to dismiss these suggestions, but losing work is painful, frustrating and embarrassing; in other words, these are among the highest-consequence UI decisions, and it is important for the UI behavior to match user expectations.

[roya0045]

Pretty much all the things listed are already implemented and you ignored them.

Stating that you are an "experienced QGIS user" and not knowing that processing methods make temporary scratch layer by default when no output is provided is already documented.

The warning when you quit is there to tell you that those layer will no longer be viable and gives you an option to save them properly.

If something is in memory, you wan save it to update its content, but it will still be in memory and get deleted.

Saving the project and saving a change made to a layer is not the same. You shouldn't blame the UI/UX elements when you are badly interpreting every text you see.

[jcpayne]

Sure, it's easy to blame the user, but user experiences tell you something about how good your UI design is. Ignoring user experience is not likely to lead to good UI design.

You wrote "The warning is there"...but you're ignoring my experience, which is that I didn't know what to do about it or what it meant. A little more information would have saved me. And yes, I am experienced with QGIS, but we all tend to use some features and ignore others, depending on what we are doing. I had done a lot of advanced analysis but very little digitizing.

"Saving the project and saving a change made to a layer is not the same." "Save" connotes saving to disk in every piece of I've ever used; you're telling me that QGIS's non-traditional approach, which violates user expectations, is good UI design? I disagree.

"If something is in memory, you wan save it to update its content, but it will still be in memory and get deleted." Does that sound like standard UI design?

[elpaso]

@jcpayne can you suggest us how to improve the warning message? Sounds like a simple improvement to me.

[ccrook]

This is a very old issue - I found the same thing long ago. There was no appetite in the community at that time for implementing into QGIS, but I did get some good suggestions around building a plugin to address this. The general agreement was that users should save data to a permanent format in whatever was the most appropriate format for their purpose. I guess the analogy would be that the new layer is like a new document opened in a word processor. Until it is saved with a name it has no persistence.

So it is a paradigm question - is the project the equivalent of a document, or is a layer the equivalent of a document. I suspect that to most users the project feels like a document.

The memory layer saver plugin that I wrote then is still available and still does the job of saving scratch layers. It is not a core plugin and does need to be installed, but once installed it does silently manage saving the layer. It creates another file alongside the project file, so the user does need to keep the two together if moving them. Not a perfect solution, but it has saved me a lot of pain!

I still think that this would most sensibly be a core feature of QGIS, at least as an option. I use scratch layers a lot (eg contour layers). The data in them is an integral part of the project and it makes no sense to not save them by default.

Scratch layers are a really useful feature. The ability to save a project and reopen it later unchanged is also a really useful feature. It would be nice to be able to do both!

[nyalldawson]

The only other suggestion I have (on top of all the existing warnings) would be to disable the "save edits" button for memory layers.

[jcpayne]

I can see that scratch layers are a useful feature.

Regarding the UI, I think the most important thing is that if you hit "Save" on a scratch layer, it should be crystal clear to the user that it is not actually saving to disk. I'm not sure what the best way to accomplish that is; maybe a popup warning that "This layer will be lost when you shut down QGIS unless you "Make Permanent". If that gets annoying you could allow the user to turn off the warning using the "Don't show this warning again" trick. (Note that "...unless you Make Permanent" is a clue to the user about where to look for solutions).

Secondarily, I did read the warning when I was shutting down QGIS, but didn't know a) what a scratch layer was or b) what the scratch icon looked like. Would it be possible to put an image of the scratch layer icon in the warning that you get when you shut down QGIS? Something like "Changes made to layers marked with this icon (XXX) will be lost unless the layer is Made Permanent (see Properties menu)".

[m-kuhn]

Since the warning is already shown anyway (read: we already have an obtrusive message box in place) there could be a list of all affected layers and an action to export those features.

[roya0045]

Secondarily, I did read the warning when I was shutting down QGIS, but didn't know a) what a scratch layer was or b) what the scratch icon looked like. Would it be possible to put an image of the scratch layer icon in the warning that you get when you shut down QGIS? Something like "Changes made to layers marked with this icon (XXX) will be lost unless the layer is Made Permanent (see Properties menu)".

I agree that the warning must be made more clear, another alternative might be to show the symbol of the memory layer beside the "scratch" in the error message, this way the user could associate both. This proposition is a good idea, though I would prevent disabling the message. Sometimes you forgot that some important layers are still temporary and this message is a good reminder at best and a minor inconvenience at worst.

I agree that alternating between the term "temporary", "scratch" and "memory" can lead to confusion.

"Saving the project and saving a change made to a layer is not the same." "Save" connotes saving to disk in every piece of I've ever used; you're telling me that QGIS's non-traditional approach, which violates user expectations, is good UI design? I disagree.

In this case the save in the main qgis app save the project, this means all the path to the layers, their symbology, their state, etc. Sometimes you don't want to make your temporary layer permanent and sometimes you don't want to save the changes made to a layer. The main button is called "save Projet" and is also located in the project menu for this reason.

Have the save button as a way to save absolutely everything could lead to some annoyance if you did not want to preserve some info, especially if you are generating a ton of scratch layer and need to discard them afterward to prevent using too much space on your system.

This is an honest mistake to make but I'm simply stating that things are the way they are for a reason.

[Saijin-Naib]

I also am surprised that the Memory Layer Saver plugin isn't in core, at least as an option, as ccrook stated. It is very important and saves a lot of headaches, especially for users migrating from ArcGIS where one can define a default.gdb and scratch.gdb that will automatically receive all permanent and temporary processing outputs without user intervention.

After deploying that plugin I've seen a lot less mishaps with data disappearing.

[m-kuhn]

The memory layer plugin has the conceptual issue that it saves data into the project file and not a proper gis data format (supporting spatial indexes etc.), that makes it unacceptable for core in its current form.

I think that the above proposal should solve the problem correctly.

[rjwillson]

the memory layer saver plugin by @ccrook is one of the very best plugins in QGIS and saves so much time in our workflows because we don't have to keep saving to a directory, naming files that may not be useful after a few days time. I have also used it to build multiple layers with preferred styles into our Q startup template so that users can start digitizing right away once they open the template. The combination of QGIS's startup templates, memory layer saver, and the more recently implemented make permanent/save scratch layer (Added by @nyalldawson) have been total game changers and have allowed me and our users to move away from ArcGIS desktop. I will never go back

additionally, when you can save persistent memory layers in Q projects it allows you share simple projects with new GIS users that may not have access to certain directories. Non-GIS users have a hard time understanding that the data is not usually stored in the project file so the persistent memory layers are a blessing.

Would it not be possible to have the memory/scratch layers saved to the project by default (i.e., build it into the core). Then give users the option to toggle a setting in the options to turn off this behavior, i.e., if they don't like to manually remove their scratch layers. That seems like the safer approach so one doesn't lose data and is much better for new users. Currently, I have to explain to our users that they should ignore the warning message when closing Q because I always have them install the memory saver plugin as all our templates are based on it working.

if the plugin was made core, the *.mht file could just be nested in the zipped package and then there would only be one file

[m-kuhn]

What's an .mht file exactly? Shouldn't that be a gpkg-inside-qgz?

[rjwillson]

it's an accessory file that gets stored alongside the Q project file once you activate Memory Layer Saver plugin and add a memory layer to project

according to the plugin author: "The memory provider data is saved in a portable binary format (QDataStream) that is saved with extension .mldata alongside the project file"

not sure which would be the better format

[m-kuhn]

That sounds pretty much what I was referring to in https://github.com/qgis/QGIS/issues/30958#issuecomment-516283858. I think we'll need a builtin container based on a well established format for project specific data. This container format can then act as a backend for "materialized" memory layers too (which we then really should call scratch layers consistently instead of memory layers).

[rjwillson]

yah having the memory layers stored in a built-in container, e.g., GeoPackage with all its benefits, within the .qgz file would be excellent.

I'm not sure what the best name for these layers would be, currently it is difficult to explain to casual or intermediate-level users that "temporary", "scratch" and "memory" layers are at times different and at other times the same thing

The key distinguishing feature that a casual user needs to know is where the data is stored, i.e., is it stored in the project file (currently stored in the mldata file with the plugin) or stored in another type of data file

currently I put the following text in my group heading so that users know the memory layers will be saved

"Will be Saved if You Save the Project - Memory/Scratch Layers are Stored in User's Q Project MLDATA File"

[nyalldawson]

I guess the danger with using any existing data format is that the conversion will be lossy. Memory providers have support for field trips which can't be reproduced 100% in any dish based format.

On the other hand, it's not ideal to make our own new custom data format...

That's why previous discussions around this topic have stalled. We need a lossless storage format, but don't want to make one!

[rjwillson]

definitely sounds tricky.

The memory layers became even more useful once you added the ability to make them permanent (which keeps the styles, joins, etc) rather than doing a save as/export which would lose the styles and joins (total game changer!). I have 20 or so empty memory layers built into our primary template so that when users make a project from the template they have access to our customized styles and joins. Then if they want to make the layer permanent (e.g., into a shapefile usually) they get to keep all the enhancements.

thanks so much for that addition!!

your thought on disabling the " . save edits" . button for memory layers could work. Currently I put the save edits and save project button right beside each other on user's toolbars and tell them to click both. If save edits was disabled for memory layers, that would save a click each time.

the portable binary format (QDataStream) that is saved with the extension .mldata alongside the project file using the plugin seems to do a good job but I admit I don't usually put layers with more than 100 records into a memory layer as a precaution. To store the .mldata file within the *.qgz file would be desirable for portability, but perhaps it is not possible.

[roya0045]

Can't we do something with hdf5, since its already in QGIS ? (if I remember correctly)

[gwideman]

And 4 years later, this is STILL an issue. It is still way too easy to create a new layer that looks just like* the other layers in the Layers panel, and yet any work you put into the layer, with conscientious efforts to save the data, vanishes on exiting the program. The quit message about scratch layers STILL fails to be clear what it is you're going to lose. Mentioning "you will lose scratch layers" is useless if you didn't know you created a scratch layer. It just seems like some unimportant internal thing that the program is fussing about, akin to the message you get on quitting from Microsoft programs about having a large amount of data in the clipboard.
The program KNOWS the names of the layers it's about to discard, so it's positively user-hostile to not spell that out in the "you are about to lose something" message. The fact that, as the OP points out, no amount of hitting save buttons, including Save Project, will save these unwantedly-scratchy layers, adds to the expectation that data will not be lost. Yes, I get that a scratch layer could be useful, but this is something that the user should create EXPLICITLY, not something that happens by default from some operations without proper notice conspicuous enough for a user who might not even know what a scratch layer is. (In my case it apparently happened because I had to fumble through the Refactor process because there's no simple function to just change the order of attribute fields, itself a rather puzzling deficiency that frustrates the ability to proceed incrementally.)

• Maybe the idea is that we are supposed to notice the little symbol to the right of the item in the Layer tree: The icon is in the shape of an 8-legged insect. Not sure how this is supposed to symbolize "scratch layer", but it does well represent that this is a bug.

Also, of course it's fantastic that this open source software exists, thanks to the extensive passion and labors of volunteers. It's just frustrating when the immense utility of the software is undercut by easily-rectified UI stumbling blocks.