Log pruning fails when 'ed' is not installed

[rfxn]

• better error output on log prunes when ed is not installed
• look at alternatives to in-place log prunes without ed (in-place as in preserving the original inode)

[sporks5000]

We could change

printf "%s\n" "1,${trim}d" w | ed -s $inotify_log 2> /dev/null

to instead be

sed "1,${trim}d" $inotify_log > ${inotify_log}_temp 2> /dev/null
cat ${inotify_log}_temp > $inotify_log 2> /dev/null
rm -f ${inotify_log}_temp 2> /dev/null

I have not looked at the internals of how sed works, but since running a "sed -i" results in an inode change, I'm assuming that it's also deleting a file and writing the entirety of the new file to disk. The above adds reading the file contents and then writing them to disk a second time, but allows us to keep the same inode.

Question: What are we trying to achieve by keeping the same inode? I know that changing the inode could break "tail -f" functionality, but I don't see anything in the project that uses "tail -f", and if there are other things that ARE running "tail -f" on these files, would it be possible to change them to "tail -F" instead?

[rfxn]

@sporks5000 Effectively that inotifywait writes to an inode; restarting it is significantly time-consuming on some systems.

[sporks5000]

Oh - I hadn't thought of that. If that's the case, the solution I presented above would potentially run a risk of instances where new lines could be written between between the original read of $inotify_log and the overwrite of $inotify_log. I don't think the risk is huge, but it's definitely worth thinking on to see if there might be another method that would prevent this.

I will keep you posted if I come up with something.

[sporks5000]

Here's the inotifywait command in the code as it is currently:

$nice_command $inotify -r --fromfile $inotify_fpaths $exclude --timefmt "%d %b %H:%M:%S" --format "%w%f %e %T" -m -e create,move,modify >> $inotify_log 2>&1 &

And here's the inotify log being trimmed in the code as it is currently:

if [ "$log_size" -ge "$inotify_trim" ]; then
    trim=$(($log_size - 1000))
    log_chars=`printf "%s\n" "1,${trim}p" | ed -s $inotify_log 2> /dev/null | wc -c`
    tlog_new=$(( `cat $inspath/tmp/inotify` - $log_chars ))
    echo $tlog_new > $inspath/tmp/inotify
    printf "%s\n" "1,${trim}d" w | ed -s $inotify_log 2> /dev/null
    eout "{mon} inotify log file trimmed"
fi

In order to eliminate our reliance on "ed" we could add the following function:

function log_inotify_out {
    local line
    while read line; do
        if [ -f $trim_inotify_touch ]; then
            local log_size=`$wc -l $inotify_log | awk '{print$1}'`
            local trim=$(($log_size - 1000))
            sed "1,${trim}p" $inotify_log 2> /dev/null | wc -c`
            local tlog_new=$(( `cat $inspath/tmp/inotify` - $log_chars ))
            echo $tlog_new > $inspath/tmp/inotify
            sed -i "1,${trim}d" $inotify_log 2> /dev/null
            eout "{mon} inotify log file trimmed"
            rm -f $trim_inotify_touch
        fi
        echo "$line" >> $inotify_log 2>&1
    done
}

And then call inotifywait like this

$nice_command $inotify -r --fromfile $inotify_fpaths $exclude --timefmt "%d %b %H:%M:%S" --format "%w%f %e %T" -m -e create,move,modify | log_inotify_out &

And trim the log like this:

if [ "$log_size" -ge "$inotify_trim" ]; then
    touch $trim_inotify_touch
fi

Because the echo statement in the function is being called separately for each line, it is re-opening $inotify_log to append to it each time it's given a new line rather than holding it open. As a result, it will not be impacted by changes to the inode that the file exists on. This does have two downsides, though:

1. Each line output from inotifywait has additional overhead associated with it. I do not have any idea how to even go about determining how much additional overhead this would be, but I doubt that it would be enough to noticeably impact anything under the vast majority of circumstances.
2. The inotify log will not trim until a new line is output by inotifywait. If the file is large enough to be trimmed, it's likely that new lines are getting written fairly regularly, but there is no real way to guarantee when it will happen. Once again, this is probably not a big enough issue to negatively impact anything under the vast majority of circumstances.

@rfxn - What are your thoughts on this? Can you think of any compatibility issues that it might run into?