Provide JSON batch job logs for cloud

[lmsurpre]

Is your feature request related to a problem? Please describe. We have a Liberty app which uses the Liberty Batch feature. We run on IBM Cloud in Kubernetes and we structure our logs in JSON and route them to console so that they get collected by the default LogAnalysis (LogDNA) agent.

        - name: WLP_LOGGING_MESSAGE_FORMAT
          value: json
        - name: WLP_LOGGING_MESSAGE_SOURCE
          value: ""
        - name: WLP_LOGGING_CONSOLE_FORMAT
          value: json
        - name: WLP_LOGGING_CONSOLE_LOGLEVEL
          value: info
        - name: WLP_LOGGING_CONSOLE_SOURCE
          value: message,trace,accessLog,ffdc,audit
        - name: WLP_LOGGING_JSON_FIELD_MAPPINGS
          value: loglevel:level,ibm_datetime:timestamp

This works for message, trace, ffdc, and audit logs, but unfortunately I can't find a way to route our job logs to console. This results in them being logged to the ephemeral disk of the containers (and potentially lost).

Describe the solution you'd like In an ideal world, I could configure the batch logs to get sent to console (instead of a file) in json format via env variable (in a similar way to the trace and access logs). However, I'd really like to capture the information from the folder hierarchy of the job logs as properties of the structured json output.

For example:

{
"jobname": "my-batch-job",
"instance": "1",
"execution": "1",
"part":"1",
"step":"step1",
...
"message":"MESSAGE_HERE"
}

Describe alternatives you've considered An alternative would be to support this type of logging through the logstash-collector feature.

Additional context Was directed here from https://stackoverflow.com/questions/63015360/is-it-possible-to-send-liberty-batch-logs-to-console?noredirect=1#comment111447583_63015360

[scottkurz]

Seems we could add another "batch" or "joblog" message source type to: <logging messageFormat="json" messageSource="message,trace,accessLog,ffdc,audit" />

as described in the doc.

Adding @donbourne @follis ..

[donbourne]

Also should be added to logging element's consoleSource and to logstashCollector feature's source attribute.

[prb112]

We're still interested in this issue. Are there any updates? many thanks for your time, Paul

[pvaennman]

Our dev-teams (@ SHB Sweden) are also very interested in seeing this feature being implemented and available, hopefully as soon as possible. Thanks in advance, Peter

[awisniew90]

Providing some use case details for SHB Sweden on Peter's behalf...

The goal of their dev teams (and I'm sure other users as well) is to stream any type of log data coming from the Liberty server via Kafka or similar ELK stack. This requires logs to have a JSON output format.

Currently this is possible for most output files in Liberty (e.g. message, trace, accessLog, ffdc, audit). Adding support for batch job logs would complete this picture.