Support custom CLI tool for formatting logs

[ebekebe]

Some of our services do structural logging in JSON. For development, we pipe stdout/stderr to a CLI tool like bunyan or pino-pretty, which parses the JSON and prints human readable log lines instead.

I would like to do the same when using skaffold dev, but as skaffold prefixes lines and outputs the logs of multiple services, this is not straight forward.

Expected behavior

Skaffold yaml:

[...]
logs:
  formatters:
    SERVICE1:
      stdout: bunyan
      stderr: bunyan
    SERVICE2:
      stdout: pino -i v

Semantics:

• Capture stdout of SERVICE1 and pass it to an instance of bunyan. Capture the output of bunyan and treat it as if it was the original output of that service. In particular, prefix it as configured.
• Capture stderr of SERVICE1 and pass it to a second instance of bunyan. Analogously to above, treat the output as if it was the original stderr output of SERVICE1.
• For SERVICE2, transform only stdout, but not stderr. Pass stdout to pino -i v and handle output like described above.
• For all output, pass through the colors (don't strip it out).

Actual behavior

There is no way of specifying a formatter.

A work-around is setting the log prefix to the empty string and pass it to pino. pino is smart enough to detect if a line is JSON and outputs non JSON lines verbatim. For bunyan, there is no such simple solution.

With the work-around, I see these drawbacks:

1. Text color is lost for non-json log lines and other output from skaffold.
2. There can only be one formatter that formats the output of all services, but it is often helpful to filter log levels or unwanted fields per service.
3. The log entries of services can't be prefixed, e.g. with service name.
4. Sometimes, e.g. in case of a container exiting, the output is prefixed even if the config doesn't allow prefixing (see example in the linked repo below).

Information

• Skaffold version: v2.0.2
• Operating system: macOS Ventura
• Installed via: Homebrew
• Contents of skaffold.yaml:

apiVersion: skaffold/v4beta1
kind: Config
metadata:
  name: skaffold-example
deploy:
  logs:
    prefix: none
manifests:
  rawYaml:
  - deployment.yaml

Steps to reproduce the behavior

1. Clone https://github.com/ebekebe/skaffold-example
2. npm install -g pino-pretty
3. skaffold dev | pino-pretty

[aaron-prindle]

@ebekebe - thanks for submitting the feature request here. From the use case you outlined above, I agree this would be a helpful feature for users. Did you want to attempt adding this functionality in Skaffold?

Also, as a potential short term workaround, skaffold has an Events API: https://skaffold.dev/docs/design/api/

You could look at the events here (namely /v2/applicationLogs) which have the raw logs and metadata (which container made the log, etc.) and write a script to parse the events and use the metadata there to make the desired log output. Lmk if that would be helpful here and I can talk more about eventing or how do this if our docs aren't enough here

[ebekebe]

@aaron-prindle

Did you want to attempt adding this functionality in Skaffold?

No, maybe at a later point. We are currently evaluating skaffold for local dev as well as CI use. Since we can work around this issue ok for now, I will focus on big road blocks first (e.g. see my PR for daemonless buildkit CI).

You could look at the events here (namely /v2/applicationLogs) which have the raw logs and metadata (which container made the log, etc.) and write a script to parse the events and use the metadata there to make the desired log output. Lmk if that would be helpful here and I can talk more about eventing or how do this if our docs aren't enough here

That is interesting. I will certainly have a look at this one. Thanks for the pointer.

[renzodavid9]

Comment for triage party.