Logs are overwritten with new pipeline run

[nsheff]

Log files are now being reset every time a pipeline runs.

Logs should be appended instead. It has worked this way for years, I'm not sure who changed this or how, but now I can no longer find logs for previous runs because they are being overwritten by the last run.

[vreuter]

@nsheff is it known whether this is the case in the released 0.14.0 version?

[vreuter]

@nsheff are you talking about the file pointed to by a pipeline manager's pipeline_log_file attribute, or something else?

[vreuter]

From first glance, one way that seems this could happen (by no means am I saying necessarily that it's happening here) is that logmuse is given an argument for logfile that matches the value of the pipeline_log_file attribute on the manager.

https://github.com/databio/logmuse/blob/00f8e9f5fd769f7b10fc51dd0ee514f851707eb5/logmuse/est.py#L255-L261

A manager opens that file in append mode with tee, but logmuse appears to open the file in (over)write mode.

[vreuter]

^^ Regardless of whether the above is responsible for what's being observed right now, it seems like... a) the logger setup in the pipeline manager needs to protect against this (definitely) -- collision b/w filepath passed to logmuse as log message destination and the manager's own logfile b) logmuse should change it's file opening mode from overwrite to append (perhaps)

[nsheff]

False alarm. I was confusing the looper log file with the pypiper log file.

Basically, I was using a looper compute template that was teeing the output. This was just doing it per run, so it overwrites on each run. The pypiper log is separate, and it was correctly appending as expected. So, there's not actually anything wrong with pypiper.

To help users realize this more quickly in the future, I'm adding the path to the pypiper log file now in the logged preamble.

[vreuter]

To help users realize this more quickly in the future, I'm adding the path to the pypiper log file now in the logged preamble.

👍 Nice!