Backup

[leotiger]

The backup process needs a revision:

1. The log does only include the latest backup reference. It's more a status file right now but not a log file
2. ZipBackup should allow to control $ignorePathsand $ignoreFolders and maybe the timeout setting, e.g. from the site configuration panel
3. The folderToZip function should add an empty folder for all folders listed in $ignoreFolders or you should think about a third array, e.g. $addAsEmptyFolders.

The second point is important, not only for a pro version. Right now the backup process can produce what I would call a "silent bug". The process does not terminate, the working indicator spins forever and the backup folder contains a process file which may consume hundreds of MB with a .zip.a233434 something extension. Adding some extra folders in code to the $ignoreFolders I was able to solve this.

Regards. uh

[rhukster]

All great ideas. PRs welcome!

[leotiger]

I do a PR this week. u

[leotiger]

Nearly done on my side... Before digging into this on the GRAV side I would like to send some screenshots and discuss this topic a bit:

1. In fact I have to do two PR, one for GRAV itself and one for the Admin plugin. As one thing depends on the other
2. Inside of the GRAV package I heavily enhanced the ZipBackup.php file to cope with several features I expect to see supported for backups: scopes (context), maxsize for the whole backup store to clean up old backups FIFO when the assigned backup store amount is topped, FIFO on max age based on days, overwriting predefined ignores, clear all existing backups inside a store, etc.
3. Admin plugin will include in my version a backup for users with admin.pages rights that invokes the user://pages scope for backup. For admins the backup button will look like the cache button allowing to select scopes for the backup. In the admin configuration the backup process can be configured and the configurations can be tested first informing about the estimated result and size. Admins can this way fine tune the process and pre-configure with 7, 8 options the backup process.

That's huge but nearly done. Is this welcome?

Regards. uh

[leotiger]

@rhukster

To give an idea here's a first screenshot:

Apart from the dropdown select for Backup to select different backup scopes and the single purpose backup for users in behalf of admin.pages right, everything's working in my test environment. My doubt: I blow the code base for ZipBackup.php by factor 10. I could send you the code on a private basis first before doing PRs that consume your time and mine. I still have to do a compatibility test with admin versions prior to the latest releases, but I had those in mind.

My code solution does not add significant load and adds not even 500ms to the process. The compensation: a highly adaptive backup scheme.

Let's speak code:

                'runInTestMode' => static::$runInTestMode,
                'currentScope' => static::$configScope,
                'backupScopes' => static::$backupScopes,
                'ignoreTopLevel' => static::$ignorePaths,
                'ignoreFolders' => static::$ignoreFolders,              
                'ignoreFileTypes' => static::$ignoreFileTypes,
                'siteIgnoreTopLevel' => static::$excludeTopLevel,
                'siteIgnoreFolders' => static::$excludeFolders,
                'addAsEmptyFolders' => static::$addAsEmptyFolders,
                'intersectSiteTopLevels' => static::$intersectTopLevels,
                'intersectSiteIgnoreFolders' => static::$intersectIgnoreFolders,
                'backupProcessTimeout' => static::$backupTimeout,
                'forceAddAsEmpty' => static::$forceaddasempty,
                'logBackup' => static::$logBackup,
                'zipFileStatus' => $zipFileStats['status'],
                'assumedCompressionRatioForTests' => static::$testModeCompressionRatio,
                'bytesExcluded' => static::$bytesExcluded,
                'filesExcluded' => static::$filesExcluded,
                'filesToZip' => $zipFileStats['filestozip'],
                'bytesToZip' => $zipFileStats['bytesReadyToZip'],
                'zippedBytes' => $zipFileStats['zippedBytes'],
                'zipFilesIncluded' => static::$zipFilesIncluded,
                'compressionRatio' => $zipFileStats['ratio'],
                'zipSavings' => $zipFileStats['saving'],
                'excludedPathList' => static::$not_included_json,
                'backupStarttime' => $backupDuration['start'],
                'backupEndtime' => $backupDuration['end'],
                'backupDuration' => $backupDuration['duration'],                
                'remainingTimeout' => $backupDuration['remaining'],
                'elapsedTimeBeforeSave' => static::$elapsedTimeBeforeSave,
                'keepDays' => static::$keepDays,
                'backupPath' => static::$backupDestination,
                'backupPathMaxStorageCapacity' => $fifoStoreStats['maxSize'],
                'backupPathCapacityUsed' => $fifoStoreStats['backupCapacityUsed'],
                'backupPathFilesDeleted' => $fifoStoreStats['filesDeleted'],
                'backupPathPurgedBytes' => $fifoStoreStats['purged'],
                'backupPathFilesExceededMaxDays' => $fifoStoreStats['purgedFilesExceededMaxDay'],
                'backupPathFilesExceededMaxCapacity' => $fifoStoreStats['purgedFilesExceededMaxCapacity'],
                'backupPathLastCheck' => $fifoStoreStats['time'],               

That's the output of the backup process reflector offering insides into what's available on my code basis.

I can offer this as a plugin but there are several restrictions inside admin and by design of GRAV there's no possibility to hook in more than one "takeover" for admin features. That's great because this reduces back-draws and overloads, but it's bad for plugin developers: There's no way to take over a admin task more than once, if I got that right from my code analysis: Admin features are closely tied to presentation and admin features do not offer many hooks to bind extra features or to substitute them. (Well, everybody is free to present a substitution for the admin plugin itself but that's not what it is about here.)

Regards. uh

[rhukster]

wow, that's an impressive set of options. I think this might be best served as an advanced backup solution as a standalone plugin, but yes, to make that work in the admin plugin, to allow that level of integration.

Do you happen to remember the places you would need the hooks/integration in admin? Would you be willing to work with me on this?

Cheers!

[leotiger]

As I told you: I integrated over ZipBackup.php in the GRAV core. For this reason I said that I would need two PRs, one for admin, another for GRAV itself. Without changing ZipBackup it's not possible to offer an admin integrated solution. I could only offer a solution that presents itself as an independent task in the left sidebar, like Data Manager for instance. From a GRAV perspective that's not simple... Backup should present itself in one place, not in two.

[rhukster]

I believe we can modify the admin to allow for a alternative backup solutions. Right now it's very hardcoded towards the ZipBackup provided by Grav itself, but really we could make that configurable as long as there is a common interface to actually call and execute the backup process.

This will require some admin plugin changes of course.

[rhukster]

Similar would have to be done for for bin/grav backup of course

[leotiger]

Ok, you got it. That's what I tried to do... to offer a more versatile interface... not everything needs to show up in admin, but it should be possible to use all of those features, or via admin, or via admin pro or third party plugins... Right now I only have one thing left: restricting the "thing" to accounts allowing backups for content by account.

[leotiger]

Another insight. Backup Scopes (Contexts):

    protected static $backupScopes = [
        'user',
        'log',
    'config',
    'plugins',
    'pages',
    'themes',
    'imagecache',
    'cache',
    'data',
    'site',
    'system',
    'clearall',
    'purge',
    'purgetests',
    ];

These are the predefined scopes in the interface apart from the default scope existing up to now and the pure site.config scope. The user callable "config" scope is not to be confused with the site.config scope, it produces a backup that includes all files in the current Grav instance root and the configuration files inside of user/config while the site.config scope runs on the pure basis of what's configured in site.config in the Configuration tab of admin.

[leotiger]

Looking at the grav cli backup command: should work out of the box like it is right now and could be enhanced to support scope calls... but cli is for "hardcore" admins and admins with access to cli may benefit from scopes but the rest of the backup universe I pushed in is not vital for them, they have the means to produce every kind of backup they want (consoling :-)): CLI admins live on both sides...

[leotiger]

This is what's needed in CLI: one function call and the capture of arguments. The function to call allows for these options: public static function prepareBackup( $excludeTopLevel = null, $excludeFolders = null, $excludeAsEmptyFolders = null, $excludeFileTypes = null, $intersectTopFolders = null, $intersectFolders = null, $ignoreCase = null, $phptimeout = null, $logBackup = null, $runInTestMode = null, $maxSpace = null, $keepDays = null,
$scope = null, $forceaddasempty = null, $testcompressionratio = null )

[leotiger]

@rhukster

Done! Ready for a first release. I will publish today or tomorrow:

Cheers. uh

[leotiger]

[leotiger]

Full CLI support with extra options via CLI is done as well. Still a 0.1.0 version... I will ping you as soon as the repository is set up completely.

u

[rhukster]

Wow that's lookin great!!

[leotiger]

test, sorry... character restrictions apply... :-)

[leotiger]

@rhukster

Here we go: 0.1.0 is out: https://github.com/leotiger/grav-plugin-backup-manager

u

[rhukster]

nice, will test this soon..

[leotiger]

Updated to 0.1.1 to fix some minor issues and add some minor improvements.