Feature Request: Back up into folders instead of Tar archives

[ads103]

I like to export my backups to an external backup repository using Borg Backup. However, because the backups generated by unraid-appdata.backup are in tar archives, I cannot take advantage of Borg Backup's deduplication features.

I'd like to have the option to back up my appdata as folders, instead of as tar archives. I'd like the result of using this option to be seeing each application's data saved in a folder in the specified backup destination, as opposed to the tar archives that are standard now.

Perhaps rsync could be used for this? Perhaps the main backup routine could resemble something like this?

    /**
     * Create a backup by copying folders using rsync, instead of tar
     * Friendly for deduplicating archivers to handle backups later
     * @param $container array
     * @param $destination string The generated backup folder for this backup run
     * @return bool
     */
    public static function backupCopyContainer($container, $destination) {
        global $abSettings, $dockerClient;

        self::backupLog("Backup {$container['Name']} - Container Volumeinfo: " . print_r($container['Volumes'], true), self::LOGLEVEL_DEBUG);

        $volumes = self::getContainerVolumes($container);

        $containerSettings = $abSettings->getContainerSpecificSettings($container['Name']);
        if ($containerSettings['backupExtVolumes'] == 'no') {
            self::backupLog("Should NOT backup external volumes, sanitizing them...");
            foreach ($volumes as $index => $volume) {
                if (!self::isVolumeWithinAppdata($volume)) {
                    unset($volumes[$index]);
                }
            }
        } else {
            self::backupLog("Backing up EXTERNAL volumes, because its enabled!", self::LOGLEVEL_WARN);
        }

        $tarExcludes = [];
        if (!empty($containerSettings['exclude'])) {
            self::backupLog("Container got excludes! " . PHP_EOL . print_r($containerSettings['exclude'], true), self::LOGLEVEL_DEBUG);
            $excludes = explode("\r\n", $containerSettings['exclude']);
            if (!empty($excludes)) {
                foreach ($excludes as $exclude) {
                    $exclude = rtrim($exclude, "/");
                    if (!empty($exclude)) {
                        if (($volumeKey = array_search($exclude, $volumes)) !== false) {
                            self::backupLog("Exclusion \"$exclude\" matches a container volume - ignoring volume/exclusion pair");
                            unset($volumes[$volumeKey]);
                            continue;
                        }
                        $tarExcludes[] = '--exclude ' . escapeshellarg($exclude);
                    }
                }
            }
        }

        if (empty($volumes)) {
            self::backupLog($container['Name'] . " does not have any volume to back up! Skipping");
            return true;
        }

        self::backupLog("Calculated volumes to back up: " . implode(", ", $volumes));

        $destination = $destination . "/" . $container['Name'] . '.rsync';

        $rsyncOptions       = array_merge($tarExcludes, ['--archive', '--relative']);    // Add excludes to the beginning - https://unix.stackexchange.com/a/33334

        self::backupLog("Target archive: " . $destination, self::LOGLEVEL_DEBUG);

        foreach ($volumes as $volume) {
            $rsyncOptions[] = escapeshellarg($volume);
        }
        $finalRsyncOptions       = implode(" ", $rsyncOptions) . escapeshellarg($destination);

        self::backupLog("Generated rsync command: " . $finalRsyncOptions, self::LOGLEVEL_DEBUG);
        self::backupLog("Backing up " . $container['Name'] . '...');

        $output = $resultcode = null;
        exec("rsync " . $finalRsyncOptions " 2>&1 " . ABSettings::$externalCmdPidCapture, $output, $resultcode);
        self::backupLog("Rsync out: " . implode('; ', $output), self::LOGLEVEL_DEBUG);

        if ($resultcode > 0) {
            self::backupLog("rsync run failed! Rsync said: " . implode('; ', $output), $containerSettings['ignoreBackupErrors'] == 'yes' ? self::LOGLEVEL_INFO : self::LOGLEVEL_ERR);
            return $containerSettings['ignoreBackupErrors'] == 'yes';
        }

        self::backupLog("Backup created without issues");

        if (ABHelper::abortRequested()) {
            return true;
        }

        return true;
    }

[Commifreak]

This is actually on the ToDo list for the same reason as yours :)

https://forums.unraid.net/topic/132721-plugin-ca-appdata-backup-restore-v25/page/18/#comment-1251220

[ads103]

Ah, that's what I get for not looking deep enough. Thank you!

[Commifreak]

The recent forum activity revealed, that dedup software like Duplicacy is able to use non-compressed tar's for deduplication (https://forums.unraid.net/topic/137710-plugin-appdatabackup/?do=findComment&comment=1365996). I dont know if Borg is able to do that as well?

If Borg is able (and Duplicacy is), I dont see any advantage of implementing this. At least not now.

Could you check if Borg is capable of this?

[ads103]

It looks like Borg's import-tar feature (https://borgbackup.readthedocs.io/en/stable/usage/tar.html) may offer similar functionality. But, it looks like Borgmatic (https://projects.torsion.org/borgmatic-collective/borgmatic) does not expose this functionality.

I'm beginning to feel like it may be more appropriate to ask the Borgmatic developers to expose the Borg's built-in tar-handling feature.

[PeteBa]

+1 for this. (also thanks for the great unraid plugin)

I use urbackup with its cross-platform support and btrf snapshotting for very quick backup times. It does file de-duplication out of the box but not block dedupe. An rsync-like option instead of tar would be great.