[Feature]: Backup Manger in the UI

[peaklabs-dev]

General Idea

Currently only databases are backed up. Also, only some databases have a restore option. See: https://github.com/coollabsio/coolify/issues/2501. Streamline database backups and offer a one click restore option without having to manually download and upload the backup. Also there should be a second checkbox that allows us to set container backups -> a container backup is just a backup of all the persistent data (volumes) of the container.

Detailed Description

I think it should be pretty straightforward for the pro @andrasbacsai. I thought about all the points and how I would implement it below, hope it helps (can not implement it myself for now as I am not experienced enough with Larvel):

• [ ] 1. Let us backup the full coolify instance to the cloud and one click restore it -> all settings... The settings are probably stored in the coolify database, so a coolify database backup would be sufficient.
• [ ] 2. Add other remote storage type for example SFTP and WebDAV, is natively supported in Laravel: https://flysystem.thephpleague.com/docs/
• [ ] 3. Add WebDav as a remote storage type -> https://flysystem.thephpleague.com/docs/
• [ ] 4. Set encryption password and encryption type -> the backups sent to remote storage should be fully encrypted if possible for security reasons Encryption is really important especially for databases. Could be done via AES-256 and a password to dycrypt
• [ ] 5. View all backups in a list. Display all backups like with database backups
• [ ] 6. One click service restore or database restore from the backup manger UI -> hit restore and it restores Download backup form the cloud to the local machine as soon as we hit restore and then restores it -> overwrites the volume and or the database or the config
• [ ] 7. Delete backups from the list manually Delete button, when hit deletes locally or on s3 or on SFTP via a remote find command
• [ ] 8. set an expiration time -> for example, after 7 days, the oldest backup will be automatically deleted from the remote storage solution. Can be done via find with a line like this:

  
  keep_backups=10

delete old Backups with find

ssh "${remote_target}" "find ${remote_dir} -type f -name '*-backup.tar.gz' -mtime +$keep_backups -exec rm {} \;"

- [ ] 9. set what to backup -> only database, only files or everything --> everything means full container, database and all configurations set in ccolify
Database already works
Files would essentially be every volume of the docker container, since all other docker data is not persistent.
Both would just let us schedule a cron job to do both.
-> As all services run in containers (even in the future when k8s is supported) there is essentially just the need to backup all volumes in the folder (but it is better to let us select backups on a container level/ service level -> just backup the WordPress volumes): 
```bash
/var/lib/docker/volumes

-> At the Coolify level, we can select Backup All --> and with Backup All, it basically just backs up all the volumes in the Docker volumes directory.

• [ ] 10. one click manual backup -> Let us click a button which triggers a full backup
• [ ] 11. only store backups in remote -> so the backups are only stored in remote, not also on the coolify instance storage (secure storage) --> After the backup is uploaded to the remote storage/ sftp or s3 delete all local copies of the backup. with -rm
• [ ] 12. button to download backup to localhost or my current device (for local backup) -> a simple download button to download the backup to my PC, for restoration of the backups wiche essential just replaces docker volumes it overwrites all fiels with the fiels of the backup
• [ ] 13. Lock backups, these backups will not be deleted from remote storage automatically can only be deleted manually -> Not idea yet how to implement but maybe move to a differente folder on the remote storage called lock which is not cleaned every time a new backup is sent to remote
• [ ] 14. Make sure to compress backups with gzip
• [ ] 15. Also let us set a folder and a subfolder in the cloud bucket where the backups should be stored
• [ ] 16. Scheduled backups -> Schedule to let us choose hourly, 12 hourly, 24 hourly... -> just use cron to schedule a backup task, like for database but for volume backup
• [ ] 17. clone a resource and let us select a backup of a volume. So we can rollback to a resource without rolling back an actively running instance. Let us clone the resource and while cloning select a backup for the volumes that will be used so we can use an old state of the persistente data to check for example if the bug is still present there.
• [ ] 18. Ability to backup all Database and all volumes in one click globally -> so basically backup everything on the server
• [ ] 19. Backup Coolify settings and application settings and so on via coolify.json

Manual script as a demo (source: https://schroederdennis.de/docker/docker-volume-backup-script-sichern-mit-secure-copy-scp-nas/)

#!/bin/bash
# # # # # # # # # # # # # # # # # # # # # # # #
#                Configuration                #
# # # # # # # # # # # # # # # # # # # # # # # #

# Directory to be backed up
source_dir="/var/lib/docker/volumes"
c# Directory in which the backups are to be saved
backup_dir="/opt/docker_backups"
# Number of backups to be kept
keep_backups=10
# Current date and time
current_datetime=$(date +"%Y-%m-%d_%H-%M-%S")
# Name for the backup archive
backup_filename="${current_datetime}-backup.tar"
# Target server information -> SFTP location
remote_user="root"
remote_server="192.168.40.50"
remote_dir="/opt/docker_backups"
# # # # # # # # # # # # # # # # # # # # # # # #
#           End of configuration            #
# # # # # # # # # # # # # # # # # # # # # # # #

remote_target="${remote_user}@${remote_server}"
backup_fullpath="${backup_dir}/${backup_filename}"

# Shut down Docker container -> Is recommended but probably not needed as it is bad to shot down production containers
docker stop $(docker ps -q)
# Create the backup archive
tar -cpf "${backup_fullpath}" "${source_dir}"
# Restart the Docker container
docker start $(docker ps -a -q)
# Compress the backup archive
gzip "${backup_fullpath}"
backup_fullpath="${backup_fullpath}.gz"
# Copy the backup to the target server with SCP without password
scp "${backup_fullpath}" "${remote_target}:$remote_dir/"
# Delete older local backups with `find`.
find "$backup_dir" -type f -name "*-backup.tar.gz" -mtime +$keep_backups -exec rm {} \;
# Delete older remote backups with `find`.
ssh "${remote_target}" "find ${remote_dir} -type f -name '*-backup.tar.gz' -mtime +$keep_backups -exec rm {} \;"

echo "Backup has been created: ${backup_fullpath} and on ${remote_target} copied."

[algora-pbc[bot]]

💎 $100 bounty • CloudGakkai

💎 $100 bounty • matt.dekok

💎 $23 bounty • Martynas Sklizmantas

Steps to solve:

1. Start working: Comment /attempt #2389 with your implementation plan
2. Submit work: Create a pull request including /claim #2389 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to coollabsio/coolify!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @peaklabs-dev	Aug 5, 2024, 10:24:26 AM	#2999

[DeVoresyah]

putting a bounty on this to make it high-priority

[balmacefa]

UI coolify_domain.you/settings#backup

Related files

https://github.com/coollabsio/coolify/blob/88581c898356e7292f999890a8aa6fff29e650ad/resources/views/livewire/settings/backup.blade.php#L11

https://github.com/coollabsio/coolify/blob/main/app/Jobs/DatabaseBackupJob.php#L30

https://github.com/coollabsio/coolify/blob/main/app/Jobs/DatabaseBackupStatusJob.php#L18

https://github.com/coollabsio/coolify/blob/88581c898356e7292f999890a8aa6fff29e650ad/app/Livewire/Project/Database/Backup/Execution.php#L10

Related blade files

https://github.com/coollabsio/coolify/tree/88581c898356e7292f999890a8aa6fff29e650ad/resources/views/livewire/project/database

[DeVoresyah]

UI coolify_domain.you/settings#backup

Related files

https://github.com/coollabsio/coolify/blob/88581c898356e7292f999890a8aa6fff29e650ad/resources/views/livewire/settings/backup.blade.php#L11

https://github.com/coollabsio/coolify/blob/main/app/Jobs/DatabaseBackupJob.php#L30

https://github.com/coollabsio/coolify/blob/main/app/Jobs/DatabaseBackupStatusJob.php#L18

https://github.com/coollabsio/coolify/blob/88581c898356e7292f999890a8aa6fff29e650ad/app/Livewire/Project/Database/Backup/Execution.php#L10

Related blade files

https://github.com/coollabsio/coolify/tree/88581c898356e7292f999890a8aa6fff29e650ad/resources/views/livewire/project/database

hi, currently coolify only supports backups for the database. instead this issue/feature request is to backup more than database

[m1daz]

If this will be completed, please allow options to choose drives on a server to backup to, not just S3

[Thijmen]

@m1daz What do you mean with drives? A local folder path, I assume?

[m1daz]

Yes, not on S3

[w7tf]

Would it make sense to allow a one click restore option? If you back up your volume data from any docker app for instance, clicking "restore" would replace the current volume with the one of the backup.

Another great option would be creating a new "resource" out of a backup. I don't know if you're familiar with Hetzner backups, but if you backup your machine, you could click "create a new machine from backup". That would spin up a new VM with the data of you backup. The exact same mechanism would come in handy with resources as well. Backup -> Create a new resource from backup -> New resource is created (with different domain and or ports)

[Thijmen]

I think it's important to have a clear goal here; is it also the goal to back up the volumes? If so, that can be a lot of data.

[w7tf]

I think it's important to have a clear goal here; is it also the goal to back up the volumes? If so, that can be a lot of data.

Indeed I'd say it is important. For stateless applications such as NextJS and Astro where you probably send your files to S3 anyways, it's not important. However for the wordpress folks where some data such as images is located outside of the database this feature would be very beneficial.

[Thijmen]

How would you determine per application what to backup? Only local volumes, persistent docker volumes etc?

[peaklabs-dev]

@Thijmen @w7tf The goal is just to refactor and improve the current backup flow and add missing features. Database restore should be available for all databases, one click restore (no manual download and upload of the backup file). Almost all docker containers have volumes that are persistent and these need to be backed up which is the point of a backup, as the container is not persistent but the volumes are. and also some other enhancements life SFTP storage option not just s3.

[peaklabs-dev]

@w7tf I added your idea as pint 17 to the feature request: https://github.com/coollabsio/coolify/issues/2389#issue-2341711069 @Thijmen

How would you determine per application what to backup? Only local volumes, persistent docker volumes etc?

There would be two checkboxes: Backup Database, Backup Container --> The backup container will backup all persistent data of the container (volumes). Updated my request for more accuracy: https://github.com/coollabsio/coolify/issues/2389#issue-2341711069

[Thijmen]

SFTP would be very easy, as Laravel supports that out of the box with Flysystem; https://flysystem.thephpleague.com/docs/

[peaklabs-dev]

SFTP would be very easy, as Laravel supports that out of the box with Flysystem; https://flysystem.thephpleague.com/docs/

I see SFTP and WebDAV and some more is supported by this that is amazing.

[peaklabs-dev]

@Thijmen If you have time to implement it, I would be more than happy to test it and make suggestions.

[m1daz]

How would you determine per application what to backup? Only local volumes, persistent docker volumes etc?

The things I would 100% want backed up are my postgres resource storage and a storage volume for another nodejs project. I also can't stress this enough, but I would really like local backups on specific drive mounts or folders on those drives, etc (since I have 8 different disks), not just remote ones. Coolify can be used for managing local servers too, not just cloud ones.

[neo3k]

Great job! What about resource-level backups? Being able to restore an application/resource individually.

[rihards-simanovics]

Ideally speaking I would love to see a backup/restore strategy similar to that of Plesk - a cPanel like control panel for hosting websites.

In Plesk you are able to backup the server configs, and separate subscriptions (domains and their subdomains) including the DB and the files.

Thing with Coolify is I think it was meant to be just for hosting the SPA and cold start applications which don't store much info other then DB perhaps and use S3 for block storage.

As you have docker containers and people can host projects such as WordPress, there needs to be a way to create three different backups per project:

1. Volume/Storage;
2. Database; and
3. Project Configs.

Config backup is covered by the Coolify DB backups; project DB backup is covered by the additional DBs - which can be added in Coolify backup settings, albeit you must add each DB manually. This means that we only need to find a way to create a volume/storage backups.

Also, restore strategy needs consideration. With Plesk for instance, you can initialise a basic instance of Plesk and configure it enough to connect to S3 storage, then just pull the latest backup; or if you need to restore only one project, then just select that one project for restore.

I think right now the best work around is to find a PaaS that provides ability to create instance snapshots/backups of the VPS, which most thankfully do, but for extra charge, or segregate your projects to different servers. Those that need dedicated backup strategy go to server X and those don't need it go to Y.

[peaklabs-dev]

As I have some time and have a quite a full understanding on what all is needed I will work on this /attempt #2389

Options

• Cancel my attempt

[picocodes]

I think this should be broken down into smaller tasks that can then be progressively merged into core.

[liamdawson]

I'd particularly love to see encryption for database backups, would be nice for that land without having to wait for the other features.

[nycterent]

Mostly interested in adding an option to backup volumes for applications that use volumes to upload data

[Paillat-dev]

I would love to put a bounty on this if it would actually mean it being done faster, but I don't feel like it's the case with the already present generous bounty.

[peaklabs-dev]

@Paillat-dev I will work on this. At the moment we are focused on releasing v4 stabel and fixing the last bugs and some perfomance enhancements.

[Paillat-dev]

@peaklabs-dev Happy to hear that. Thanks to you all contributory for this truly amazing project !

[wowtah]

Love Coolify so far, I am considering it for a production scenario, but it would need some more backup/restore features for this to be a safe and viable option. So backups for databases, volumes/mounts, everything that changes basically. More widely supported targets/methods (like (S)FTP, SCP) would also have my preference over S3.

So following this thread with great interest :)

[AurelienConte]

I'm looking to use coolify for a homelab server and i would love to see the support for rclone (https://rclone.org/). He is providing a large variety of storage and this will be very cool for people with low budget.

[gitekDev]

like wowtah (backup => SFTP )

[litobro]

Adding my support for encryption on backups - the whole point of self-hosted is being in control of your data, pushing unencrypted data to an S3 store in the cloud can be dangerous for many reasons.