System logs field

[buchanae]

In Funnel, we've found that there are a lot of logs coming from the system and not the executor (container) that are important: an error message from the object store client, an error message from the system about a disk being full, a docker run command, etc. We've been avoiding adding this to TES and putting it in separate endpoints, or accessing these logs via external systems such as rsyslog or Kibana, but I think it's worth proposing here. It's an optional field, and in the logs message, so not much overhead to implementations.

[buchanae]

Closes #73

[briandoconnor]

Do you need STDERR and STDOUT? I like the idea of collecting these and making them accessible for the reasons you say. You may need to specify in the comments of the schema that this may be subject to a line limit for a given implementor:

http://editor.swagger.io/#/?import=https://raw.githubusercontent.com/ga4gh/workflow-execution-schemas/0.1.0/swagger/proto/workflow_execution.swagger.json

[buchanae]

Do you need STDERR and STDOUT? I like the idea of collecting these and making them accessible for the reasons you say. You may need to specify in the comments of the schema that this may be subject to a line limit for a given implementor:

These logs are not tied to any single process, so stdout/err don't fit. These are any logs the system decides are relevant outside of the actual executor/docker process. I've improved the field docs in the latest commit, adding some notes about the (lack of) specification around the format/size/etc.

[geoffjentry]

👍

[pditommaso]

👍

[buchanae]

I've added documentation to state that the system logs are only available in the FULL task view, which is consistent with task stdout/err logs and other fields which are potentially more than a small string.