Contemplate another log that only has the run info

[balarsen]

I have been using GNU parallel a lot and find --joblog to be super useful.

The format of those is

A logfile of the jobs completed so far can be generated with --joblog:

  parallel --joblog /tmp/log exit  ::: 1 2 3 0
  cat /tmp/log

Output:

  Seq Host Starttime      Runtime Send Receive Exitval Signal Command
  1   :    1376577364.974 0.008   0    0       1       0      exit 1
  2   :    1376577364.982 0.013   0    0       2       0      exit 2
  3   :    1376577364.990 0.013   0    0       3       0      exit 3
  4   :    1376577365.003 0.003   0    0       0       0      exit 0

This is a lame example as the command is just exit but it puts the full command line there. I like the time and size arguments a lot. The send and receive are if we are running on other machines, which I doubt we will do.

Proposed enhancement

A run log would be really useful for debugging and providence of files. Especially for failures.

Closure condition

This issue should be closed when: either a decision to not do it or an implementation.

[jtniehof]

So basically just a log of every command line run, when, and how long it took? Thinking of this as a table or another log file?

A different approach (not necessarily better) would be to impose more formatting on the text logs and provide tools to parse them, so you could pull out this information or come up with some other report. That would split the "logging of information" and the "determining what information is interesting in what context" tasks.

[balarsen]

Agreed, I have done similar on the past.

Thoughts on tools actually existing and who are they are for either the pro who setup the chain or the user monitoring it. They have different needs.

The current logging is well intentioned but not well utilized.

[jtniehof]

logwatch is supposed to be fairly generic but you're basically writing a bunch of Perl. I actually had a student write something to parse Python logs that we can potentially adapt/extend for this purpose; it was for a logwatch-like summary (so I get emailed the summary daily.)

To really do it right we probably would have to do a big review of the logging setup and have standards for the messages to make them easier to parse, but that wouldn't necessarily be a bad thing in itself. In practice if I have a task that needs digging into the logs I get a cup of coffee and the man page for grep....