Exclude patterns GUI improvements

[m3nu]

Just noticed that Nextcloud (desktop client) comes with a nice list of default exclusions. Maybe something for Vorta to assimilate.

I agree that one simple kind of patterns would be enough.

Originally posted by @m3nu in https://github.com/borgbase/vorta/discussions/838#discussioncomment-369487

[erAck]

BackInTime also has a sensible set of default excludes:

• .gvfs
• .cache/*
• .thumbnails*
• [Tt]rash*
• .backup
• *~
• .dropbox*
• /proc/*
• /sys/*
• /dev/*
• /run/*
• /etc/mtab
• /var/cache/apt/archives/*.deb
• lost+found/*
• /tmp/*
• /var/tmp/*
• /var/backups/*
• .Private

[m3nu]

We discussed adding defaults before and the feedback wasn't good due to the large variety of OS and use cases.

So if we add defaults, they will be OS-specific and disabled by default.

[m3nu]

Suggested by @Wikinaut in #970

I cannot find that borg option --exclude-caches to be used by vorta. Was that intended?

See https://borgbackup.readthedocs.io/en/stable/usage/create.html?highlight=exclude%20caches

I suggest to add this as a vorta option which is active by default.

[m3nu]

Getting close to starting this feature. Was thinking about this a bit more and would want those features:

• Maintain templates of exclusions and enable them for selected profiles. Like "Dev Cache Files" or "App Cache Files".
• Currently I use 2 groups of exclusions: specific paths (e.g. ~/Library/BlahBlah/Cache and patterns that should be matched everywhere (e.g. */.git)
• Maybe we also want "Exclude folders with this file"
• Add description to exclusion
• Import and export exclusions as CSV? (pattern, desc)

Anything to add? @Hofer-Julian @bastiencyr ?

[Hofer-Julian]

Sounds good. One idea:

• Add description to exclusion
• Import and export exclusions as CSV? (pattern, desc)

How about allowing comments with #? In that case you don't need an extra field for descriptions and the export format would be a list of patterns and comments instead of a csv.

[m3nu]

I like the idea of making the import/export file a valid patterns file.

In the UI, I'd prefer to have pattern and commend separate. Else we literally have a free text box again.

[bastiencyr]

Souds good too.

Maintain templates of exclusions and enable them for selected profiles. Like "Dev Cache Files" or "App Cache Files".

For naming, maybe some more meaningful names for users like "User Temporary Files", "Dev Irrelevant Files". Its more user orientated and may not know what a cache is.

[m3nu]

macOS via arkTanlis on Reddit.

(Some wildcards used here are not appropriate for the default fnmatch patterns we use.)

~/.Trash
**.DS_Store
~/Library/Caches
~/Library/Logs
**.cache
~/.npm
~/Library/Containers
**.git
**node_modules

[m3nu]

Adding my own excludes:

*/.DS_Store
*/._sync_*
*.pyc
*.mp4
*.mkv
*/.~*
*-journal
*/.owncloudsync.log
*/.cache
*/__pycache__
~/Pictures/20*
~/Pictures/Phone

[m3nu]

Some more inspiration from Back in Time. They probably keep a list for Linux.

[real-yfprojects]

I sketched a first GUI mock-up. What do you think?

[m3nu]

I think it's great. 🙌 Not much to add.

So the data model is like description, pattern, OS, on/off? Some of those are built-in and some are supplied by the user? We should put the built-in patterns in a reusable format, since they may be relevant for other projects, if we end up collecting enough.

[Hofer-Julian]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

[real-yfprojects]

So the data model is like description, pattern, OS, on/off? Some of those are built-in and some are supplied by the user? We should put the built-in patterns in a reusable format, since they may be relevant for other projects, if we end up collecting enough.

There patterns that can be grouped together with a name. They can be toggled through the GUI or by commenting them out in raw text. The predefined pattern groups that have tags (if they are app, os specific,...) and possibly a description. There is a search bar for searching through the patterns/groups by group name and tags as well as pattern contents.

We can integrate online sources for patterns like gitignore.io or github/gitignore for excluding different development files.

[real-yfprojects]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

I had a look at Pika Backup. It seems like they only allow excluding existing files/folders that can be added to the exclusion list through a file chooser.

[m3nu]

That's not very flexible. E.g. to exclude a type of files.

[Hofer-Julian]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

I had a look at Pika Backup. It seems like they only allow excluding existing files/folders that can be added to the exclusion list through a file chooser.

That specific feature is not released yet. But that screenshot should give you an idea: https://gitlab.gnome.org/World/pika-backup/-/issues/94#note_1504221

[m3nu]

Yeah, having multiple patterns in a group makes sense too.

So the data struct would be like: group description, [list of patterns], OS (as per Python sys.platform), on/off

[real-yfprojects]

Yeah, having multiple patterns in a group makes sense too.

So the data struct would be like: group description, [list of patterns], OS (as per Python sys.platform), on/off

Yes, that would be a good start. However the user should be able to toggle single patterns, edit the predefined patterns and add own patterns. This could be realised in two database tables: PatternGroup and Pattern. However my idea was getting the data from a raw text file/entry the user can edit.

That specific feature is not released yet. But that screenshot should give you an idea: https://gitlab.gnome.org/World/pika-backup/-/issues/94#note_1504221

Thanks. I like that the mockup shows the size of the excluded files.

[fooness]

Hello everyone. Could you please point me to the current list of different exclude defaults/suggestions? Maybe I missed the link here, or did search for the wrong keywords. I thought I could look the exclusions up here in the repo, without installing Vorta (as I’m basically using borg without GUI for my macOS backups).

---

Here’s what I came up with for now; it’s a mixture of Arq’s exclusion list as well as the suggestions from the previously linked reddit thread and from the posts in this very GitHub issue … comments and feedback very much appreciated; this is macOS-specific.

**/.*.swp
**/.DS_Store
**/.VolumeIcon.icns

# not quite sure what the following two exclusions are for
**/.file
**/.vol

**/.DocumentRevisions-V100/
**/.MobileBackups/
**/.MobileBackups.trash/
**/.Spotlight-V100/
**/.TemporaryItems/
**/.Trash/
**/.Trashes/

**/.cache/
**/.fseventsd/
**/.npm/

**/Library/**/Cache/
**/Library/**/Caches/
**/Library/**/Containers/
**/Library/**/CrashReporter/
**/Library/**/DerivedData/
**/Library/**/Developer/
**/Library/**/Metadata/
**/Library/**/MobileSync/
**/Library/**/Logs/

**/iTunes/

**/build/
**/dist/
**/node_modules/
**/tmp/
**/vendor/

/System/
/Users/Shared/
/Volumes/
/bin/
/cores/
/dev/
/etc/
/home/
/macOS Install Data/
#/nix/
#/opt/
/private/
/sbin/
/tmp/
#/usr/
/var/

# comment: /nix/ + /opt/ + /usr/ maybe should not be excluded due to e.g. homebrew …

[m3nu]

Thanks for sharing! We don't ship a default list yet. Still work in progress. Based on the current plan, we may do different groups that can be toggled. Since exclusions are really OS-dependent and personal.

[fooness]

Based on the current plan, we may do different groups that can be toggled. Since exclusions are really OS-dependent and personal.

Yes, the definitely are; been looking for the macOS-specific list.

We don't ship a default list yet. Still work in progress.

That’s the part I obviously have missed reading through this issue; will keep an eye on future updates.

PS: I’ll be doing some more research on this matter, and in case of any new insights, I’ll be editing/updating my list above.

[diivi]

I made a design for the exclude GUI, it's available here and is awaiting community feedback. I'll make changes to it based on the response received. Please let me know what can be improved! I tried to keep it visually pleasing and beginner friendly, while still giving the user some powerful options. Attaching a quick screenshot here too:

• Presets can be selected in the Add Dialog to make this a little less confusing for beginners (Simple by default, powerful when needed.)
• Multiple files and folders can be added at once instead of clicking the add button again and again.
• Comments can be added to the file/folder being added, these will show up in the Raw tab and will be ignored in the Patterns tab.
• If user selects a preset, the patterns field becomes disabled.
• Raw tab will show all the rules, kinda like the current pattern exclusion field, but will include comments and have functionality to import/export a set of rules as txt, as @Hofer-Julian mentioned.
• Inspired by mockups made by @real-yfprojects, but I removed the separate patterns and presets tab, as I thought this might be simpler for most users (I couldn't differentiate between the two tabs and too much functionality at once seemed overwhelming)

[erAck]

Looks good in general.

If user selects a preset, the patterns field becomes disabled.

I'd rather display the preset's content then and have it editable. For that, the Pattern text edit should be larger and popup resizable.

What should happen if a(nother) preset is selected while patterns (in the main dialog) already exist? Replace, after warning? Merge (add without duplicates)? Needs further buttons? Or shouldn't rather a preset choice only be presented if there are no excludes yet?

What does the bottom field with /tmp/my_file tell? What file is that? The file the excludes are saved to? And what does the circled checkmark indicate? File name is valid? Or would it be a single pattern's edit field? Unclear.

The Add pattern to exclude popup shouldn't have a Save button, because that's not what it would do (or would it? confusing), it would add the pattern(s), so maybe Add instead? The only real Save is on the main dialog. (I hope).

Nitpicking: File names starting with # are valid file names, even # with a blank following, even more they might be wanted to be excluded. How to resolve the conflict?

[diivi]

Thanks for the detailed review! Very helpful.

I'd rather display the preset's content then and have it editable. For that, the Pattern text edit should be larger and popup resizable.

Yes, I can do that, afterall there's no harm in making them editable (in the field) and it adds functionality.

What should happen if a(nother) preset is selected while patterns (in the main dialog) already exist?

I was thinking if a pattern present in a preset or a pattern in general already exists in the patterns tab, it would not be replaced. So yes, for a mixed case, merge. I don't think a warning in the UI would be necessary when merging though.

What does the bottom field with /tmp/my_file tell?

Ah, agreed that it's not clear what this does, but do you think replacing the placeholder text with "Enter a path pattern to be matched" helps (I just missed changing the placeholder text). Explanation - /tmp/* was excluded and /tmp/my_file was entered, hence the check mark. This is taken from the mockups @real-yfprojects shared earlier.

The Add pattern to exclude popup shouldn't have a Save button, because that's not what it would do

True. Will change this, thanks.

File names starting with # are valid file names, even # with a blank following, even more they might be wanted to be excluded. How to resolve the conflict?

Good point, for now I can just think of removing the option to add comments in that field and keep that only for the Raw tab. Open to other ideas though.

[real-yfprojects]

I added some comments on figma. Some design choices I would like to discuss:

What about having a preset list instead of a checkbox? This will make it much more comfortable to look through and select a preset.

In the current mockup one isn't able to determine which rules an added preset defines without opening the raw tab.

I somewhere read about dropping the raw tab. I like it since it allows power users to add/edit patterns fast as well as copy and pasting them. I would especially appreciate the option to add comments so that I can remember what a pattern does when I come back to a profile after some time. One could even have a syntax for defining custom groups (e.g. #%% group_name) so that the graphical list doesn't get to long and cluttered.

File names starting with # are valid file names, even # with a blank following, even more they might be wanted to be excluded. How to resolve the conflict?

See borg docs:

The --exclude-from option permits loading exclusion patterns from a text file with one pattern per line. Lines empty or starting with the number sign (‘#’) after removing whitespace on both ends are ignored. The optional style selector prefix is also supported for patterns loaded from a file. Due to whitespace removal, paths with whitespace at the beginning or end can only be excluded using regular expressions.

So whitespace trimming should also be done by vorta automatically whenever the user adds/edits patterns.

What does the bottom field with /tmp/my_file tell?

Ah, agreed that it's not clear what this does, but do you think replacing the placeholder text with "Enter a path pattern to be matched" helps (I just missed changing the placeholder text). Explanation - /tmp/* was excluded and /tmp/my_file was entered, hence the check mark. This is taken from the mockups @real-yfprojects shared earlier.

This was the solution I came up with for checking whether one configured excludes correctly. I suggest the tooltip shows Check a path for exclusion. On entering vorta will try to match the path with all patterns. If a matching pattern is found a check mark is displayed else a cross. The check mark could be implemented using QToolButton, it would be clickable and highlight the matching pattern in the pattern list.

Yes, I can do that, afterall there's no harm in making them editable (in the field) and it adds functionality.

The preset should change to None automatically when editing a preset.

[diivi]

I added some comments on figma.

Checked and made some edits.

What about having a preset list instead of a checkbox? This will make it much more comfortable to look through and select a preset.

Can you share a sample UI that I can refer to? Is it something like this? (changing the active list item will repopulate the text field contents, and editing the text field will make it switch back to "None") I agree this might be easier than opening the combobox again and again to view different presets. Will make the add dialog a bit bigger though, but I guess that would be ok.

In the current mockup one isn't able to determine which rules an added preset defines without opening the raw tab.

Is that information necessary for the user? I feel like the presets should just be there to assist the user in getting pre-defined patterns. If they really want to see why a pattern is there they can just look in the Raw tab. I wanted to keep the Patterns tab less cluttered and the simplest thing a user would want.

[real-yfprojects]

Is it something like this?

Yes, exactly.

I feel like the presets should just be there to assist the user in getting pre-defined patterns. If they really want to see why a pattern is there they can just look in the Raw tab. I wanted to keep the Patterns tab less cluttered and the simplest thing a user would want.

When adding a preset do want to show only the preset name in the pattern list or an entry for each pattern in the preset?

[diivi]

When adding a preset do want to show only the preset name in the pattern list or an entry for each pattern in the preset?

I was thinking an entry for each pattern in the preset, to let me disable singular patterns too. But if you would like the preset to be there as a group, maybe we can list all the presets first, and then singular patterns?

[real-yfprojects]

I was thinking an entry for each pattern in the preset, to let me disable singular patterns too.

Ah, that wasn't clear since your mockup didn't contain patterns only.

But if you would like the preset to be there as a group, maybe we can list all the presets first, and then singular patterns?

I think there will be a lot of patterns pretty fast. There should be a way to organize them.

[diivi]

Ah, that wasn't clear since your mockup didn't contain patterns only.

Can you check now? I went back to the approach of keeping patterns and presets separately, with a line separating them. To view all patterns associated with a preset, you can view the raw tab. This will only work when the preset is added as is, without modification, otherwise the preset becomes None and all patterns are added separately.

[real-yfprojects]

I went back to the approach of keeping patterns and presets separately, with a line separating them. To view all patterns associated with a preset, you can view the raw tab.

There should be a way to highlight the patterns in the raw view.

[diivi]

Highlight which patterns? And what kind of highlight? Highlight after search (planning to do that), or highlight = selecting text (can do that).

[real-yfprojects]

Highlight which patterns?

The patterns associated with a preset. Else you have to search them manually.

[diivi]

I made some more progress, thanks to suggestions from @real-yfprojects and @erAck , this is what the UI looks like currently: Again, the designs are available here for anyone to comment/give feedback - https://www.figma.com/file/Y70aJFOpJ1VT59lWzUZEoQ/Exclude-GUI?node-id=0-1&t=yONNiNodanw9dnE4-0.

[m3nu]

You should consider the data structure before doing the GUI.

First there could be groups that enable multiple rules. Like macOS cache files. Precise content would be hidden and may change during updates. So having presets as simple text and then putting everything in a large text field isn't the best way.

Then the user may add some raw rules. Those would be merged with the rules we ship as presets.

Presets could be just checkbox + name + details as tooltip or collapsible.

Custom rules could be what you have as patterns now. Text + checkbox.

Raw is useful to quickly add many existing rules.

Finally, Preview would merge it all together to help debugging.

[diivi]

• Okay, I'll change the current Patterns tab to Custom and make that accept only custom exclusion rules.
• I'll add a presets tab that has all the grouped rules, and will be similar to the Patterns tab, but will have preset names instead of rules and a tooltip to show the details. Or, I'll make it collapsible to view all excluded files (I can do this on Figma, but will need help in making a collapsible list on qt).
• Raw remains the same
• The add dialog will be a single input text field to add a single custom exclusion rule?

Finally, Preview would merge it all together to help debugging.

Do you mean there should be another tab? What will this tab show? I think the check input below is sufficient for debugging. Also, the raw view will already show all the rules inside a preset group.

[m3nu]

Preview tab could show all rules combined how they are passed to Borg later.

Raw would be just for user input. If you combine both, how do you know what the user edited and what's from our own presets?

[diivi]

Okay, and how do you recommend the separation of user added rules and vorta preset rules be done in the preview tab?

[diivi]

Also, I'm concerned that having four tabs and too many features may be counterproductive, as it could make the interface more difficult for novice users to navigate. I agree with your first point about the presets getting updated and a simple text field might not be the best solution. But I'm not so sure about adding so many things :/. Maybe we can just change the way presets are handled and added?

[real-yfprojects]

I like @m3nu suggestion. What about having two tabs: Patterns and Raw. The pattern tab shows the patterns and collapsible pattern groups including added presets. The raw tab shows the custom patterns at the beginning and the preset patterns appended to the end. However the section of the textedit with preset patterns is greyed out and can't be edited. There won't be any merging of preset and non preset patterns, borg can handle duplicates.

[diivi]

Cool, if that's what we decide, then I'll edit my design to accomodate groups and single rules both in the patterns tab. What about the add tab? Do you think I should change something? Maybe disable the textfield when a preset is selected? Or, with the current approach:

• if it is edited then change the preset to none and treat them as seperate rules
• if an unedited preset is added, we can keep track of it and update it easily.

[vv111y]

This is great that you are doing this. While researching I found this repo for list of excludes for linux /home with rsync, it should be good for this feature https://github.com/rubo77/rsync-homedir-excludes

[diivi]

Thankyou! This is really helpful :)

[diivi]

Thanks to everyone who participated in this thread!

The current iteration after community review and feedback from the maintainers can be found here - https://www.figma.com/file/Y70aJFOpJ1VT59lWzUZEoQ/Exclude-GUI?type=design&node-id=0-1&t=vXY1HiJ152wU6FhF-0.

Attaching screenshots here too:

This is one of my GSoC projects, and I'll start coding in a few weeks. Please let me know if anyone has other suggestions/improvements that I can look into before that!

[m3nu]

Looks good and what I imagined really. Minor thoughts:

• For the Preview tab it should be clear that the text isn't editable. Else I imagine it's the same as the Raw tab.
• The text box at the bottom with "Placeholder" is for testing? Writing a full path may not be feasible and we need a file selector there? Or leave out the feature for now.
• Custom tab: We want to add a comment/notes for custom patterns?
• Search feature: not sure that's useful. Maybe just in the Custom tab for filtering out lines? Not sure how easy it is to select text in Qt in a textbox.

[sammcj]

It might be good to include a field that you can provide exclusion lists to, it's not currently clear if the UI supports .lst files or not.

e.g, the vorta equivalent of running borg create --patterns-from=borg-exclude-macos.lst

Where the .lst contains items such as:

# Siri
! **/Library/Assistant/SiriAnalytics.db
! **/Library/Assistant/SiriSyncItems.db
! **/Library/Assistant/SiriVocabulary/sv.db

# Logs and Monitoring
- **/Logs/

[real-yfprojects]

The current mockup doesn't allow patterns afaik but only excludes. There is a TextEdit for pasting in your own excludes e.g. from a different file.

[diivi]

Hey, I have started work on this feature and will soon get started with the Presets tab. These presets will be shipped by Vorta and you'll just have to select some presets and all patterns included in that preset will be excluded from the created archive!

I need some feedback/suggestions for these presets though, the format I need is -

Preset Name Description /pattern1 /pattern2 ...

Preset 2 Name Description /pattern ...

So if anyone has a grouped list (or even one set of patterns), like this - https://github.com/borgbase/vorta/issues/907#issuecomment-1482763027 it'd be helpful to get things started on this feature. Thanks!

P.S. you can add your preset by opening an issue after this feature is shipped too. These presets will be a json file in the Vorta codebase itself.

[m3nu]

Parsing this will be a minefield and we may want more metadata. Would use a dict/json format.

name, desc, os, rules format, list of actual rules, maybe?