MySQL image - Logs are not being logrotated properly and growing to abnormal sizes

[g7morris]

Issue description

The mysql.log is growing to large sizes ranging from 13 to 46 GB if left unchecked.

Questions / Pseudo-code solutions

Q1: I guess the broader question is, does this behavior impact all logs per se?

Q2: Is logrotate a thing on Docker?

Q3: @br2490 @shauntru Do we want to consider this below as a solution or the ones in the next section below?

• This looks like the easiest fix here: https://sandro-keil.de/blog/logrotate-for-docker-container/ for the various containers NOT just mysql

  • In essence, create a logrotate profile and then simply add it to all appropriate containers. I can come up with the list.

  • The questions I have with respect to this potential solution, are:

    • Does that path and service still exist on a Docker container?

    • A quick highly unscientific Google survey appears to be yes but still not clear from a best practice standpoint, if one uses logrotate "traditionally" or does one set logging higher up with json edits to the docker-compose.yml file or even Docker itself?

Additional resources / ideas

• https://medium.freecodecamp.org/how-to-setup-log-rotation-for-a-docker-container-a508093912b2

• https://success.docker.com/article/how-to-setup-log-rotation-post-installation

• Sidecar option probably the least easy https://hub.docker.com/r/blacklabelops/logrotate/

• Logrotate in general https://serversforhackers.com/c/managing-logs-with-logrotate

Please note: I didn't find that it was a challenge until several types of conditions occurred all at once, e.g. several large scale ingests with enough time passing on several different systems, so this would have only been caught in testing if we had large scale data sets and more time.

Steps to reproduce the issue

1. Use a running ISLE v1 or v2 system for 1-2 weeks
2. Ingest beyond 10 GB
3. Check log size

What's the expected result?

• that the logs would be < 1GB

What's the actual result?

• large sizes ranging from 13 to 46 GB if left unchecked. Also host servers can be taken offline if the OS HD is large enough.

[DiegoPino]

Hi, my 2 cents IMHO rotating logs (or even keeping them) via Docker container specifics feels like a bad ideal I would say the better behavior of exposing them directly as > /dev output of the container (so you can see them using docker logs -tail etc ) seems the better approach and in that case managing/cleaning them becomes a host level task (can be crond-ed) which can be automatized to clean/remove or archive them. Since we all agree on data separation when planning Docker containers, maybe there are other better ideas on how to deal with them? Issue we had is that once you expose those internal container logs to the outside via > /dev/bla, the actual storage (by default) starts happening in the docker-machine, at least in Mac (OSX). And then you have to extract the location via a docker info, then log into the docker machine! (first time i had to do it in my life) and then remove them from there. Annoying!

Also, finally, should we have different log setups (using ENV) that switch based on production / dev environment (with production outputting only ERRORS?)

Diego Pino Navarro Metropolitan New York Library Council 599 11th Av. New York, NY 10036

[g7morris]

Rotating logs is no longer handled in ISLE since release version 1.2.0. Closing ticket.