Extremely large logs because of $logger->debug in RsyncSource

[kachkaev]

Symfony production environment usually has a fingers_corssed logger, which dumps all log messages to a file only if there is an error.

During a backup, an error may occur when syncing to a remote server, e.g. if the connection is weak. Because of RsyncSource, the logs may become terribly long.

Here is the cause:

$process->run(
            function ($type, $buffer) use ($logger) {
                $logger->debug($buffer);
            }
        );

In my project, I rsync a directory with about 10,000 small files and then send the archive to a remote folder. The cron job runs every 5 minutes. If the upload fails, the fingers_crossed logger calls the nested one and appends thousands of lines to a file in app/log. Thus, some of the recent daily logs grew up to 800 MB! :–)

I know that you can disable debug messages in the nested logger like this, but I'm not sure this a good solution:

# app/config_prod.yml
monolog:
    handlers:
        main:
            type:         fingers_crossed
            action_level: error
            handler:      nested
        nested:
            type:  rotating_file
            path:  "%kernel.logs_dir%/%kernel.environment%.log"
            level: warning # <------ instead of debug
            max_files: 10
        console:
            type:  console

Seeing debug messages for failed app request is really handy, so disabling them globally is undesired.

What problem does the detailed logging of rsync solve? In which cases may it be helpful?

[kbond]

Interesting! You're right, I don't see any problem being solved which isn't already solved by:

if (!$process->isSuccessful()) {
    throw new \RuntimeException($process->getErrorOutput());
}

Are the logs generated by the above exception huge?

I'm thinking to just removing the debug calls from S3CmdDestination, ArchiveProcessor and RsyncSource. I can always optionally (flag passed during construction) add them back later if a need arises.

[kachkaev]

I haven't faced problems with rsync itself, so I can't say what the log looks like when the process is unsuccessful. It probably should be just a short error message, not a giant dump of every little detail.

I agree that just removing those calls will be enough for now. You can always add some flag to the config later on, but right now I can't imagine a case when this could be useful. Simple is good.

[kbond]

Fixed in https://github.com/kbond/php-backup/commit/e396e090e34276bdfbbb35b30d80cf94d938d3af (v1.2.0)