search

ramirojoaquin / vestacp-borg-incremental-backups

A series of bash scripts to perform incremental backups of Vesta Control Panel users and server config, using Borg Backup as backend.
GNU General Public License v3.0
23 stars 15 forks source link

Backups should not use gzip #4

Closed joshbmarshall closed 4 years ago

joshbmarshall commented 4 years ago

I have just found this project and am interested to try it but will propose some changes from some testing I've done of my own using borg to backup vestacp. If you use gzip, the rolling deduplicator of borg is made ineffective. If you use gzip --rsyncable it isn't so bad but still quite an impact compared to not at all. If you don't compress the data (particularly mysql dumps) then the rolling deduplicator of borg is very good. Borg has modern compression built in and is only used if the block hasn't been seen before, so it will actually use less cpu and network and desk if you remove gzip. I'll look into it and try to get you a pull request.

ramirojoaquin commented 4 years ago

Thanks for your suggestion Josh. To be honest i did not try it without gzip. But what you say what you say makes perfect sense.

joshbmarshall commented 4 years ago

Hi I've just been working around the VestaCP (and HestiaCP) bug where user restores don't properly set the MariaDB 10.4 authentication... I have got some changes to suggest that will improve this, are you open to pull requests? I can help with the integration for HestiaCP as well, am interested to move to that platform

ramirojoaquin commented 4 years ago

Yes of course, i am not an expert, this is my first and only project in github. But please submit the pull requests. I also move to Hestia, and i was planning to suggest a way to integrate this script collection with Hestia web ui. The Hestia devs are agree with that. I dont know if this could be a hestia plugin or integrated to the core.

joshbmarshall commented 4 years ago

Sounds like they want it in core :) I am at the moment putting together a modification to pool the users data together in a borg repo - so that multiple users with the same platform e.g. Drupal, Wordpress etc will get deduplicated

ramirojoaquin commented 4 years ago

The first version was that way. One big repo. But in practice it was not convenient. If you have a few users is not a problem, but when you have more than 500 (i am managing a big server with that amount of users), it turns very slow. The restore script was taking too long to restore a single website and its more difficult to purge a user backup. Also if for some reason there is an error (i had some), its better to have different repos, so the error only affect one user. But maybe could be an option like "use one repo for all users" in the config file. Let me know what you think.

joshbmarshall commented 4 years ago

Ah ok I can see how that's different use case. purging a user backup is straightforward with --prefix filter but it doesn't get around the speed issue. Leave that how it is. I have the database backing up directly into the borg repo (doesn't take up space on the user disk space) so will work on restorations and send a pull request