mattiash / docker-pino-sample

Sample code for logging with pino from a docker container
https://mattias.holmlund.se/2019/01/logging-with-pino-from-docker/
4 stars 2 forks source link

Does not work with node:12-alpine image #4

Open rossanthony opened 4 years ago

rossanthony commented 4 years ago

This is not necessarily an issue, just looking for some clarification on a few things based on my specific setup.

I'm using the node:12.18.0-alpine image and noticed that the entrypoint.sh script doesn't work, it produces the following error: "env: can't execute 'bash': No such file or directory"

Changing the header of the script to #!/bin/sh get it to execute the script, but it errors on L18: "bin/sh syntax error: unexpected redirection". Seems like the redirection on L18 of entrypoint.sh is a feature specifically available in bash and not in sh.

I'm curious what the > > is doing in this command - I've used single > in commands before when sending output of a script to a file for example, but haven't seen double with a space like this before...

node build/index.js > >(./node_modules/.bin/pino-pretty -t) &

I noticed that pino recommends here in their docs to use pipe | to offload the transport to a separate process from the main node app. Does < < redirection have the same effect, but additionally allow for catching of the SIGTERM and ability to pass it along to the node process?

I.e. i'm wondering if there would be any performance difference between this command:

node build/app.js | node build/utils/logTransport.js

and:

node build/app.js > >(node build/utils/logTransport.js) &
worker="$!"

Note: in the example above logTransport.js is a custom log formatter, doing a similar job to pino-pretty in the example in this repo.

mattiash commented 4 years ago

Yes, there is a difference between

node build/app.js | node build/utils/logTransport.js

and

node build/app.js > >(node build/utils/logTransport.js) &

The first command will exit with the exit-code of logTransport.js whereas the second command will exit with the exit-code of app.js. Both commands run two separate processes and pipe the output from app.js to the input for logTransport.js.

This repository is the result of reading a lot of stack overflow, bash documentation and manual testing. It might be possible to achieve the same thing with sh, but I don't know how.

I have been using alpine at $work for running node in the past, but we ended up running into too many corner-cases where alpine was slightly different from anything based on glibc that we ended up switching our containers to use debian instead. If I were you, I would use debian. If you want to keep using alpine for some reason, you can always install bash in your alpine based container.

rossanthony commented 4 years ago

Thank you for the quick reply @mattiash. I've managed to get it working by installing bash as you suggested, by adding RUN apk add --no-cache bash to the Dockerfile.

Regarding the corner-cases you mentioned you encountered while using alpine, we've actually been using alpine in production for quite a while in ECS. So far we haven't come across anything out of the ordinary. I'm curious how did the issues you ran into manifest? Sounds like we should consider switching to use the debian based node images, instead of the alpine.

Btw the example setup in this repo has been very helpful. I stumbled across it via the link you posted on this pino issue. Our app, deployed to ECS, is currently not receiving SIGTERM events. I noticed this after adding a process.on('SIGTERM') event listener and a graceful shutdown delay, however currently it never gets triggered when ECS drains running instances, during deploy of a new task for example. In my efforts to debug why this was the case I came across that issue on pino and realized that it's because of the entrypoint CMD in the Dockerfile:

CMD node index.js | node logTransport.js

Turns out the app will never receive the SIGTERM and will just shutdown, because Docker runs this type of command as shell rather than docker exec.

Sorry, one more question for peace of mind before I make the switch, have you been using the CMD ["entrypoint.sh"] method with the stdout > > redirection to another process in a production app for a while without any issues?

mattiash commented 4 years ago

Most of the problems that we encountered with alpine had to do with running software written in C++ where performance was very important. We also needed newer versions of nginx that were not available in the stable distribution of alpine. None of the problems except the performance issue with our own C++ software were really deal-breakers and we were able to solve them one by one but after a while we realised that we were spending way too much time dealing with problems that resulted from our choice to use alpine and we decided that the small size of the docker image was not really worth it for us.

I am using this method to process logs for a number of different projects with daemons that are running 24/7 logging to papertrail and it's been working flawlessly since I did the last commit to this repo 17 months ago.

rossanthony commented 3 years ago

Thanks @mattiash - reassuring to hear this has been tried and tested in a prod environment for a significant period of time.

One other question, do you know why your entrypoint script returns the sigterm code back here after it's waited for the process to gracefully shutdown? I was thinking of doing this instead:

# SIGTERM-handler
term_handler() {
  if [ $worker -ne 0 ]; then
    kill -SIGTERM "$worker" # send SIGTERM to node process
    wait "$worker" # wait for the node app to shutdown gracefully
    exit $?; # send exit code of node process back
  else
    exit 0;
  fi
}

Also wondering why L9 uses quotes: wait "$worker" but L21 is wait $worker, is this because one is inside a function and the other is not?

mattiash commented 3 years ago

The term_handler is run if the process received a TERM signal, i.e. if it was killed by a TERM signal. The exit code from a process shall reflect if it was killed by a signal. So the way I understand exit codes, the only possible exit code when you receive a TERM signal is 143.

I think wait $worker and wait "$worker" are equivalent as long as $worker does not contain spaces (which it doesn't).