Only log outside the container, not inside

[nichivo]

Currently the container writes logs to /var/log/lighttpd/ and can consume a lot of disk space over time as there is no logrotate process scheduled to run.

This patch writes server and traffic logs to stdout only to avoid consuming disk space.

[spujadas]

Cool workaround, but not comfortable getting rid of the logs altogether within the container. Closing this PR, I'll add logrotate and set up log rotation instead.

[nichivo]

Ok, but it could get messy - if you plan on using cron to run logrotate, you'll need to choose a process manager, configure services, etc. And remember to add VOLUME /var/log/lighttpd to the Dockerfile.

I think log output from the containers should be sent elsewhere (the default docker logs -f, journald, logstash, cloud service, etc), so keeping the logs inside as well seems wasteful.

Ideally, containers should mutate as little as possible (but there are always exceptions like pid files). docker diff shouldn't show many changes to the filesystem.

Have a think about it - interested to see where it goes ;)

[spujadas]

Great food for thought there, thanks!

I’m weakly opinionated on this topic so definitely open to discussion.

My initial use case was just to have a container with a light HTTP (😉) server for low key client-side testing and debugging, so didn’t need a logging strategy, let alone a long-running one. In practice if something goes wrong I just look at the logs output by the container, or jump into the container and look at the individual log files if there’s a lot to sift through.

With this trivial use case out of the way, in a production setting where logs are first-class citizens, I can see logging from apps in containers going two ways:

• Logs are streamed from the app to stdout (e.g. as per the 12 factors), there are no log files in the container (which stays relatively immutable), and a separate log management service collects and manages the logs.

  My primary concern is that with this container specifically, there are two types of logs (access and error), so by dumping everything to stdout indiscriminately we’re losing some information (namely, the type of each log record). (My secondary concern is that in my non-production setting I need an additional service/container if I want to look at the log history.)

• Logs are stored in files, and - in production - either the logs are collected from the container’s output (so same as above), or an additional agent/service collects and forwards the logs from the individual log files.

  This approach solves the two concerns from the first approach, but it has the impacts and downsides that you mentioned.

Having said that neither of the two approaches seems fully satisfying (and I may be missing other approaches that would solve all these problems 😄), so it may ultimately be down to individual preference (and of course what the local dev/ops practices require).

Anyway let me give this some more thought!

[spujadas]

Yeah I'm going to go with the simple option as per your PR, to keep everything lightweight.

[nichivo]

Great, thanks!

Yep, it can be difficult to contort "pre-Docker" apps to run inside containers and sometimes sacrifices need to be made! As for stderr logs going to stdout, I think it's worth losing that as it's possible to use better tooling if required. e.g. access logs are usually well formatted so Logstash can isolate those, and any other logs that don't match that format are therefore errors or service messages.

But it looks like this merge wiped out PR#6 - maybe this PR needed to be rebased before merging or something?!? ;)

[spujadas]

The wipe-out was caused by fat-finger conflict resolution whilst attempting an auto-merge using GitHub 😅 — has now been corrected.

Thanks for the interesting discussion on this PR!

[nichivo]

Nice!

It's good to see the most popular lighttpd image now stops quickly and gracefully :)

time docker stop lighty
docker stop lighty  0.15s user 0.03s system 38% cpu 0.467 total

and is light:

docker diff lighty
C /run
A /run/lighttpd.pid

Thanks.