Feature request: export log file

[moeller0]

Maybe we could implement something where we can force the export of the logs? Say a script that sends a signal to the main script on the reception of wg hich the script will export the current two log files into an archive with a timestamp from the export request? That way users that experience an anomaly or interesting/weird behaviour can catch a relevant log easily for post-hoc analysis?

[lynxthecat]

Sounds like an excellent idea. Are you thinking something within the overall cake-autorate or something external? Would this only be in interactive mode with console output or also external script?

[moeller0]

Not sure about the details, just thought the functionality might be nice. Maybe implement an export_current_log() function within triggered by a signal, and a helper script to send that signal with some sanity checking so we will sent to a running autorate instance....

[lynxthecat]

How about something like this:

export_log_file()
{
        (($debug)) && log_msg "DEBUG" "Exporting log file to /tmp/cake-autorate-$date.log"
        printf -v date '%(%F-%H:%M:%S)T'
        cp /tmp/cake-autorate.log /tmp/cake-autorate-$date.log
}

maintain_log_file()
{
        trap "kill_maintain_log_file" TERM EXIT
        trap "export_log_file" USR1
... 
}

And then anything within the script can trigger export with kill -USR1 $maintain_log_file_pid and we could also write out that pid to /tmp/cake-autorate so a user could also trigger export with own script, etc?

[lynxthecat]

So this could be triggered either within cake-autorate or even outside cake-autorate using another script or utility.

[moeller0]

Yes, conceptually it might be nice to offer that as an instruction on the front page. Or we add a script that issues this command and compressed the resulting files (and maybe the config file as well) into a debug archive?

[lynxthecat]

So whilst we think about scripts should I make the export location and/or filename configurable? At the moment I just have it output to /var/log/cake-autorate-$date.log (contains time too).

In any case now cake-autorate could, for example, export log files for which some parameter has exceeded a value e.g. sustained bufferbloat or something.

[moeller0]

Mmmh, well make the script take an output path override as argument and default to something sane.

[lynxthecat]

OK I'll add a path option in config with default "" and check if that is set and if so use it else use /var/log/cake-autorate-$date-time.log.

The trap on USR1 allows export to be instantiated from external script on running cake-autorate instance by the way. But export location is right now set in advance (can't pass export location with USR1 signal - would have to read in from file). But that's surely fine for now.

[moeller0]

But export location is right now set in advance (can't pass export location with USR1 signal - would have to read in from file).

Fair enough, the exporter script can simply grab it from the default location and move it to the requested location?

[lynxthecat]

Right now I output to filename with date in it. But I was thinking if we want external script it might be easier to allow override to set simple fixed location. Make sense?

[lynxthecat]

Like this:

# *** ADVANCED CONFIGURATION OPTIONS ***

# cake-autorate facilitates triggering an export of the log file either within or outside cake-autorate
# namely, to trigger a log file export:
# send a USR1 signal to $maintain_log_file_pid: "kill -USR1 $maintain_log_file_pid"
# $maintain_log_file_pid can be read from /var/run/cake-autorate/maintain_log_file_pid
# the log file export location is /var/log/ and by default the filename is: cake-autorate-$datetime.log
# or, alternatively, set this override filename here:
log_file_export_filename_override=""
log_file_export_compress=1 # compress the exported log file with its default/override filename using gzip

export_log_file()
{
        printf -v datetime '%(%F-%H:%M:%S)T'
        if [[ -z $log_file_export_filename_override ]]; then
                (($debug)) && log_msg "DEBUG" "Exporting log file with default filename to: /var/log/cake-autorate-$datetime.log"
                cp /var/log/cake-autorate.log /var/log/cake-autorate-$datetime.log
        else
                (($debug)) && log_msg "DEBUG" "Exporting log file with override filename to: /var/log/$log_file_export_filename_override"
                cp /var/log/cake-autorate.log /var/log/$log_file_export_filename_override
        fi
}

maintain_log_file()
{
        trap "kill_maintain_log_file" TERM EXIT

        trap "export_log_file" USR1
        ...
}

[moeller0]

If you make the path path of "log_file_export_filename_override" users can even steer the output to different directories (e.g. a mounted drive with sufficient storage to collect multiple files)...

[lynxthecat]

I initially did that - do you think I should? I am worried that's dangerous to allow writing to any location? So then I changed just to fixed location: "/var/log" but with override filename? I also still need to add optional compression using gzip.

[moeller0]

Well, the config script is a bash script that gets executed as root already IIRC so the user can hose their system already beyond all recognition if they so desire. I would simply document this in the help text and pass the responsibility on to the local user...

So I think it would make sense to also store the config file as well. Maybe create a directory with the data, copy log both logfiles and the config file and then create a tar.gz from that directory and delete the directory there after?

[lynxthecat]

OK I have this for now:

# *** ADVANCED CONFIGURATION OPTIONS ***

# cake-autorate facilitates triggering an export of the log file either within or outside cake-autorate
# namely, to trigger a log file export:
# send a USR1 signal to $maintain_log_file_pid: "kill -USR1 $maintain_log_file_pid"
# $maintain_log_file_pid can be read from /var/run/cake-autorate/maintain_log_file_pid
# the default log file export path is: /var/log/cake-autorate-$datetime.log
# or, alternatively, set this override path here:
log_file_export_path_override=""
log_file_export_compress=1 # compress the exported log file with its default/override path using gzip and append .gz to export filename

export_log_file()
{
        printf -v log_file_export_datetime '%(%F-%H:%M:%S)T'
    if [[ -z $log_file_export_path_override ]]; then
            (($debug)) && log_msg "DEBUG" "Exporting log file with default path: /var/log/cake-autorate_$log_file_export_datetime.log"
            log_file_export_path="/var/log/cake-autorate_$log_file_export_datetime.log"
    else
            (($debug)) && log_msg "DEBUG" "Exporting log file with override path: $log_file_export_path_override"
            log_file_export_path=$log_file_export_path_override
    fi

    if (($log_file_export_compress)); then
        gzip -c /var/log/cake-autorate.log > ${log_file_export_path}.gz
    else
        cp /var/log/cake-autorate.log $log_file_export_path
    fi
}

I think the config export can wait until we figure out scripts. Since script can just pull out /root/cake-autorate/cake-autorate-config.sh anyway?

[moeller0]

OK, you go for a fixed name but configurable path.

I had simply used something like: log_file_name_stem="/var/log/cake-autorate-" and would have allowed the user to change the whole thing to their desire... but your solution works as well. However change this from _override to the normal way of configuring the target to simply the export_log_file() function a bit.

I still like the idea of including the config file into the archive though... (even though most relevant parameters are also logged in the log file itself).

[lynxthecat]

So to be clear I've gone back to specifying and respecting full path override so now it will just write out to user override path (stem + filename).

So by default it exports to: /var/log/cake-autorate_$datetime.log

And else writes out to: $log_file_export_path_override (full path including filename).

My rationale for the above is that it could be helpful for something within cake-autorate to save out logs at particular times and hence unique filenames (e.g. prolonged bufferbloat or stall) for monitoring such events. And alternatively it could be helpful for an external script to trigger an export on running cake-autorate instance and know exactly where the file will get exported to for further processing/renaming etc. Actually perhaps both would be a good in idea.

Yes thinking about it could use USR1 for the default datetime export and USR2 for the alternative set export? This would accommodate both without forcing choice of one or the other?

Yes maybe I'll go for this since it seems to make sense. Unless you have any objections.

Also if compress is set to 1 it will copy out using gzip (no intermediate file) and append .gz to the default or alternative path.

Hope this makes sense.

[lynxthecat]

A cool idea is dawning on me here. We can set a variable like skip_shaper_rate_changes and then the user can have this running in the background with fixed rate and trigger export log whenever bufferbloat is detected. So they see if fixed rate is OK or any connection problems. That'd be pretty cool? Could optionally write out syslog warning.

[moeller0]

We can set a variable like skip_shaper_rate_changes and then the user can have this running in the background with fixed rate and trigger export log whenever bufferbloat is detected.

That still can result in potentially unbounded file growth, not a recipe for stability. But yes, for interested and cooperating (read taking care of the stored archives) users that could be sweet.

However, I think we are already doing well above the required by the cool new flight-recorder type logging, we really just need to instruct the users how to export/exfiltrate the log files on encountering odd behaviour (we should always export the current and the old log fie, as log rotation might have happened between noticing an anomaly and actually getting around to issue the export signal).

[lynxthecat]

we should always export the current and the old log fie,

Ah yes good point!

[lynxthecat]

Well this has got fairly advanced now:

# *** ADVANCED CONFIGURATION OPTIONS ***

# cake-autorate facilitates triggering an export of the log file either within or outside cake-autorate
# namely, to trigger a log file export:
# send a USR1 or USR2 signal to $maintain_log_file_pid: "kill -USR1 $maintain_log_file_pid"
# $maintain_log_file_pid can be read from /var/run/cake-autorate/maintain_log_file_pid
# a USR1 signal will trigger an export to path: /var/log/cake-autorate_$datetime.log
# a USR2 signal will trigger an export to the path set in $log_file_export_alternative_path below
# in either case both the current .log and previously rotated .log.old (if it exists) will be exported
log_file_export_alternative_path="/var/log/cake-autorate_export.log"
log_file_export_compress=1 # compress the exported log file with its default/override path using gzip and append .gz to export filename

export_log_file()
{
    local export_type=$1

    case $export_type in

        default)

            printf -v log_file_export_datetime '%(%F-%H:%M:%S)T'
                (($debug)) && log_msg "DEBUG" "Exporting log file with default path: /var/log/cake-autorate_$log_file_export_datetime.log"
                log_file_export_path="/var/log/cake-autorate_$log_file_export_datetime.log"
                ;;

        alternative)
            (($debug)) && log_msg "DEBUG" "Exporting log file with alternative path: $log_file_export_alternative_path"
                log_file_export_path=$log_file_export_alternative_path
            ;;

        *)
            (($debug)) && log_msg "DEBUG" "Unrecognised export type. Not exporting log file."
            return
        ;;
    esac

    # Now export with or without compression to the appropriate export path
    if (($log_file_export_compress)); then
        gzip -c /var/log/cake-autorate.log > ${log_file_export_path}.gz
        [[ -f /var/log/cake-autorate.log.old ]] && gzip -c /var/log/cake-autorate.log.old > ${log_file_export_path}.old.gz
    else
        cp /var/log/cake-autorate.log $log_file_export_path
        [[ -f /var/log/cake-autorate.log.old ]] && cp /var/log/cake-autorate.log.old ${log_file_export_path}.old
    fi
}