Pino for dockerized application

[Jokero]

You suggest to use transports for additional logs processing (formatting, sending somewhere, etc.). But how to use it inside docker container?

I can do the following in Dockerfile:

CMD node index.js | some-pino-transport # shell form

And it will work. But application will not be able to handle signals like SIGTERM because it will not receive them. This is because docker runs this command in shell.

CMD can be written in JSON form, but honestly I don't know how to write a command with pipe this way.

Maybe you know some workarounds?

[mcollina]

The best way is to use a docker agent that pull all the logs from all your containers and send them to a specific location. Ideally your containers should not be aware to where their logs are being sent.

[Jokero]

Yes, but this is too complicated way especially if I need just to format logs. So using node index.js | some-pino-transport looks pretty nice, but does not fit the container architecture (but maybe it does)

[mcollina]

I'm lost. Why do you need to format logs within the containers?

[Jokero]

I use Google Kubernetes Engine which collects logs from containers stdout. So the simplest and more logical solution in this case is to format logs on container level.

Two ways to achieve this: 1) use something like node index.js | pino-stackdriver // there is already module for this https://github.com/advinans/pino-stackdriver 2) somehow format pino logs on application level // maybe use custom stream which will format logs and pipe them to process.stdout

I prefer first one but can't manage to make it work now :smile:

[mcollina]

You should use the approach described in https://github.com/pinojs/pino/blob/master/docs/API.md#metadata. Let me know if this makes into an OSS module, I’ll be happy to review.

[Jokero]

Yeah, this will help. But Symbol.for('needsMetadata') is not necessary at all, because additional fields lastLevel, lastMsg, lastObj, lastLogger don't have all needed information.

For example, you will not be able to get object from child logger using lastLogger.

logger.child({ name: "child" }) // you can't get this object from lastLogger

That's why it's necessary to parse chunk string and prepare new object, then again serialize it:

var instance = pino({}, {
  write: function (chunk) {
    const parsed = JSON.parse(chunk); // or any other parser
    const formattedObject = format(parsed); // your specific formatter
    const newMessage = JSON.stringify(formattedObject);
    console.log(newMessage); // the same as process.stdout.write(newMessage);
  }
})

[Jokero]

And regarding mentioned pino-stackdriver module. It currently does not support this programmatic usage (format method), but maybe I'll sent them PR or create new module

[jsumners]

(I have not tried this.)

It looks like you can start a Docker container and give it access to a specific device (--device=/dev/foo). Which should mean you can pass it named pipe on the host system. Your application could look for the special device and, if present, create a WritableStream instance to it and supply that stream to Pino.

[Jokero]

Sounds very tricky :smile: And anyway I have to apply formatting to logs somehow. It can be done on device itself, but again, it's over-complicated.

I think I will go with this (the simplest solution):

var instance = pino({}, {
  write: function (chunk) {
    const parsed = JSON.parse(chunk); // or any other parser
    const formattedObject = format(parsed); // your specific formatter
    const newMessage = JSON.stringify(formattedObject);
    console.log(newMessage); // the same as process.stdout.write(newMessage);
  }
})

[davidmarkclements]

@Jokero your simplest solution defeats the point of Pino: parsing/formatting/reserializing in-process is likely to significantly impact performance. If perf isn't high priority, then you might want to try Winston which has a many more options in terms of formatting and log handling.

[jsumners]

Closing for inactivity. Re-open if you have further questions on the subject.

[elyas-bhy]

@davidmarkclements @jsumners Could we at least add some guidelines / best practices to the documentation on how to use Pino with containerized applications?

[jsumners]

@elyas-bhy by all means, submit a PR.

[nicokaiser]

I guess we have the same problem: We run several services in Docker containers, and the logs should go somewhere (e.g. Logstash via UDP). Some of them might use Pino in the future and some are legacy containers which write directly to Logstash in one way or another. So we need a way to pipe the output of the Pino containers (and only of those) to a pino-transport. This could be done in the container itself, but as @Jokero states, this is not very elegant...

[mattiash]

I have an example of how to log with pino from docker in https://mattias.holmlund.se/2019/01/logging-with-pino-from-docker/

[albertkang]

@mattiash, using your solution if my node process exits, i.e., process.exit(1) - the entrypoint shell does not exit and I'm left with a container is still running. How can I change your script to exit the shell as well?

[mattiash]

@albertkang I have filed an issued at https://github.com/mattiash/docker-pino-sample/issues/1. Let's continue the discussion there.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.