search

splunk / docker-logging-plugin

Splunk Connect for Docker is a Docker logging plugin that allows docker containers to send their logs directly to Splunk Enterprise or a Splunk Cloud deployment.
Apache License 2.0
63 stars 25 forks source link

Logs are being written to disk #66

Open Carles-Figuerola opened 4 years ago

Carles-Figuerola commented 4 years ago

We have just found that all the logs sent to splunk are also logged to disk.

[root@hostname docker]# sudo docker plugin ls
ID                  NAME                           DESCRIPTION             ENABLED
a6269368de77        splunk-logging-plugin:latest   Splunk Logging Plugin   true

[root@hostname docker]# pwd
/var/lib/docker/plugins/bd27a6269368de776e3773cf02ceb74cca39e4de3cca7dadc30f832f83ac37ab/rootfs/var/log/docker

[root@hostname docker]# du -sh *
93M     0da81cda054cd9f9b76278b71c743149aafb60fa89ffb4b1daeb137f6052f630
38G     0eb8c9c87335831629655535bce1bf9eb3556ef772744a66ab75f3d43219a845
80M     0f86ee162622f6be17d04fbeb3880d6920b91acaee609f418137a946fcc3c1c9
128M    17403b83b0486836c5115f0c7f549997ea33ff28a418c47b102564b5888b845c
7.7M    39b31d58d1a0da89e8af52bb7efdafb6cfc590fee72186f83cea60f12bcfcb6d
38G     3e3986be739abb8d3903e6e8c5a92e82f6ff32e3bbaf073f546875ce219ce1ef
35M     73096342216ba7ae59448e4fceced401c9c8c3cf9f91d9b94dfb2fa2a8226aca
84M     a41a0c56de2ad9ae83aa985901290b2b3d606c5a01196697c209cd63d9399af1
119M    ae61bf126a10587879b21d91c4a72cdb04a2d6762ebebf90cb617bc176f57de4
12G     dd0661c47c0ece1a02f65b045ed52f6ed475e2957d4cc8ff5d4a165fab28161a
22M     e3269b321d5f6aac6b96ec631ae348c6cca84085658d5c83ddba5fd41777f1b9
25M     e36384dd11bef199e1122657e39ac2033d2dee8b9f6d0c12805489f283d73e17
142M    ecb83fa0102dba193e4acc6f1a7f5441c2560a0b73fe0f67c1529f8565fbe384
276M    f3560b57b96ef1044d441b82f7651de398fa1509867dd661d06ceb295b5e2c6e

[root@hostname docker]# file 0da81cda054cd9f9b76278b71c743149aafb60fa89ffb4b1daeb137f6052f630
0da81cda054cd9f9b76278b71c743149aafb60fa89ffb4b1daeb137f6052f630: ASCII text, with very long lines

The logs persist even through docker service restart or even when the containers are stopped and deleted from the server.

There isn't any indication on the readme for this plugin that the logs would be written to disk as this should be a streaming plugin to send the logs elsewhere.

gp510 commented 4 years ago

Hi @Carles-Figuerola , there is a Docker environment variable called SPLUNK_LOGGING_DRIVER_JSON_LOGS that controls this logging. Set it to "false" to only send the logs to Splunk.

Carles-Figuerola commented 4 years ago

hey @gp510,

Many thanks for the prompt response. I changed the environment variables for docker:

# cat /proc/$(pgrep dockerd)/environ | tr '\0' '\n' | grep JSON_LOGS
SPLUNK_LOGGING_DRIVER_JSON_LOGS=false

But I still see the plugin /var/log folder increasing in size and I can still docker logs <sha>

Is there anything else I'm supposed to do other than adding the environment variable and restarting the docker daemon?

ursweiss commented 4 years ago

Same issue here. I expected the JSON logs are cleaned up after a container was removed, but that doesn't seem to be the case. So the option is quite dangerous and this should be mentioned in the readme. Probably the default of the ENV var should be "false" too.

How can old logs be cleaned up? Can they simple be deleted?