Infinitely growing container log file

[karabelnikov]

@bnhf

Scott, hi! Recently I discovered one problem related to the log file of our container.

One day I couldn't connect to the OpenVPN server, While studying the problem, I saw that the docker and containers and a couple more system services didn't start. I have a virtual machine with 1 core, 1 GB of RAM, 16 GB of ROM. At first I couldn't figure out what was wrong and randomly found out that there wasn't enough space on the disk. I started to find out the culprit and calculated that along the way: /var/lib/docker/containers/ there is a file ID-container-json.log (Something like this: 1518202f1c5a5cb47c446ffe5387ad85c10bb88e95c0006485a050a0db592521-json.log) has a volume of about 10 GB. Due to the fact that this file was infinitely increasing, I ran out of space and the server stopped working. After installing the container and deleting the file, everything became normal. But this file is constantly increasing in size.

Can you check it yourself? We need to figure out why it is growing so endlessly and limit its size, since this should not happen.

[bnhf]

@karabelnikov

Greetings!

This is actually more of a Docker issue, but clearly something we want to figure out. This article appears to be on point with a fix involving setting up log rotation for Docker. Looks fairly straightforward:

https://www.howtogeek.com/devops/how-to-clear-logs-of-running-docker-containers/

[karabelnikov]

@bnhf

I figured out the problem and how to solve it!

Problem: The log size of the "openvpn-web-gui" container is constantly increasing.

Reason: there is a meta tag in the code of the main page:

<meta http-equiv="refresh" content="XX">

I have it for 10 seconds, you seem to have 60 seconds by default. If you do not log in to the web interface, the log size will not increase. If you go to the web interface and leave the tab open, then every XX seconds the docker writes the main page data to the log file. That is, every XX seconds of page refresh, information is written to the log file and it increases in size (information from the main page).

Solution: add docker-compose.yml to the file of your Stacks the following code:

logging:
      driver: "json-file"
      options:
        max-size: "100k"

Firstly, you install the logging driver as a "json-file", secondly, limit the size of the log file to max 100 KB. When the max size is reached, the file is reset to zero and starts recording information again.

And then restart your Stack in Portainer.io via the "Update the stack" button.

The solution suggested Scott is the same, but it only works for newly created containers. And for existing content, the logging rule is /etc/docker/daemon.json does not work.

[karabelnikov]

@bnhf

I suggest updating docker-compose.yml file in the repository so that by default the log file does not exceed 100 kb. Auto-updating of the page is a necessary function, but over time it can lead to an increase in the size of the log to such a size that if the ROM of the virtual machine is small, the server will stop working.

I suggested PR.

[bnhf]

@karabelnikov

Nice work my friend! This is a good solution, and I agree it should be incorporated into the standard/recommended docker-compose.yml. I recently started running my own OpenVPN Server in a Proxmox VM that has fewer resources than my previous bare-metal host, so this issue would have caught-up with me soon -- thanks for sorting it out before that happened. :-)

[karabelnikov]

@bnhf

Thank You, Friend. Always in touch! I have a project that was put into production on a VMware ESXi hypervisor for 13 people. A small company. For me, this is the main tool for secure remote access.

We will continue to monitor the progress of the work further. I'm closing the assignment.