prune with prefix

[kewlfft]

If several hostnames are saved in the same repository starting their archive names with {host_name}-, then for pruning, it is necessary to add {host_name}- as a prefix with the --prefix option.

See in Borg quickstart, the prefix is said to be "very important". Using prune without prefix in emborg, I inadvertently pruned other hostname's archives. Then, one could argue it is more efficient to create one repository per server:

• there is no risk of having failure due to simultaneous backups,
• the local cache of checksums does not need to be rebuilt every time another server makes a backup.

However this prevents deduplication across servers and when space is the priority having one single repo makes sense.

Some notes for the implementation:

1. Having prefix as a command line option is dangerous, if one forgets it with simply emborg prune, it results in the pruning of the repo without filtering.
2. --prefix is used for pruning and listing, having a general prefix option in the repo settings that applies to both will prevent listing any other archives, which can be detrimental. In this case we could offer a --ignore-prefix option that ignores the prefix provided in the setting, it could apply to list and even prune: emborg --ignore-prefix list will list all archives ignoring the prefix in settings.
3. Last, we could have a prune_prefix and a list_prefixoption, this makes it a bit more complicated but addresses the points.

I would go for 2) which sticks as much as possible to Borg naming of options and logic.

19 is a first step for implementing 2) which should not create any issue, the next step could be to create the '--ignore-prefix' and extend the option to list as well. However this simple option addition already reaches the goal for safe pruning.

[KenKundert]

I have incorporated your pull request, but I cannot help but thinking we could do better. Shouldn't emborg automatically set prefix for the user? That would require splitting the archive setting in into two settings, perhaps _archiveprefix and archive. Or perhaps archive can be converted into the prefix and I just always add {{now}} to the end of whatever is specified. What do you think?

[KenKundert]

If not specified, the archive setting is now derived from the prefix setting. If neither is specified then prefix defaults to '{config_name}-'.

[kewlfft]

• prefix Yes it makes sense to go further with the prefix. Then, if I understand well, this could be a breaking change as names will be different without user's intervention in the settings. Just something to be careful about.

• suffix As far as suffix are concerned, I personnally use {{now:%Y%m%d}}, if implemented it would be good to have the option and not necessarily have {{now}} at the end.

[KenKundert]

Kewl, With the new version you can simply not specify either archive or prefix, in which case you get:

archive <-- '{config_name}-{{now}}'
prefix <-- '{config_name}-'

Or you can specify just prefix, in which case you get:

archive <-- '<prefix>{{now}}'

Or you can specify both archive and prefix. For example, you might specify:

archive = '{config_name}-{{now:%Y%m%d}}'
prefix = '{config_name}-'

If you only specify archive and not prefix, you would get the old behavior where prefix is not used in the prune command.

Conceivably I could implement a now or now_format setting, but it does not seem necessary as you can get what you want by specifying both prefix and archive.

-Ken

On Sun, Sep 22, 2019 at 11:29:54PM -0700, kewl wrote:

• prefix Yes it makes sense to go further with the prefix. Then, if I understand well, this could be a breaking change as names will be different without user's intervention in the settings. Just something to be careful about.

• suffix As far as suffix are concerned, I personnally use {{now:%Y%m%d}}, if implemented it would be good to have the option and not necessarily have {{now}} at the end.

-- You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub: https://github.com/KenKundert/emborg/issues/18#issuecomment-533974184

[kewlfft]

Understood, and I agree let's keep it simple no need for more parameters for the now format. The prefix will be applied for pruning and not for listing if I understand well, I am fine with that or with another solution but this is just to make sure I get the current implementation.

[KenKundert]

I have now extended the use of prefix to all the commands that support it, specifically check, delete, info, list, mount, and prune. I spent some time refining my changes and the documentation, but I believe I am done for now. So I recommend that you pull the latest version (1.4.4) from github.

[kewlfft]

very good, then is it now still possible to list all the archives of the repo without the prefix?

Also, I would have thought that the default prefix would be the hostname: a config can be copied on several hosts and backup on the same repo using their hostname as a prefix without overlapping.

[KenKundert]

I can add a --all option that blocks the --prefix option, meaning that list would list all archives. Is this something that would be useful for all commands that use --prefix. Seems like it would be.

I can change the default prefix to hostname.

[kewlfft]

Yes I think -all would be useful for all commands with prefix, to be used with caution for pruning but this is user's responsibility, the ability to do it makes sense.

default prefix: to make sure I am not making a wrong interpretation, I have checked again the Borg quickstart and they use '{hostname}-' as an example. It makes sense to use it this way as a default.

Use the prune subcommand to maintain 7 daily, 4 weekly and 6 monthly archives of THIS machine. The '{hostname}-' prefix is very important to limit prune's operation to this machine's archives and not apply to other machines' archives also: borg prune \ --list \ --prefix '{hostname}-' \ --show-rc \ --keep-daily 7 \ --keep-weekly 4 \ --keep-monthly 6 \

[KenKundert]

Okay, I will add the --all option.

On the default prefix, Borg does not have the concept of a config, so I don't think we can use the Borg documentation too literally. I am beginning to think that the safest thing to do is to use the following as the default prefix: '{host_name}-{user_name}-{config_name}-'. That way there is almost no chance of a conflict.

[KenKundert]

Okay, I have updated emborg to version 1.4.5. I implemented the --all option as --include-external (I reserved --all for the mount command) and I changed the default prefix so that it includes the host name, the user name, and the config name.