The json log files rotated by Docker still consume the disk

[supereagle]

Configuration for log driver : --log-driver --log-opt json-file max-size=2m --log-opt max-file=5

The rotated json log files are deleted by Docker, but the fd of these files are still kept by Docker, so their disk size can not be released otherwise restart the Docker.

The number of kept fd is very large, if it larger then the config --default-ulimit nofile=131072 of Docker, the Docker can not work normally.

# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | wc -l
11405
# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | grep 48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931 | wc -l
4882
# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | grep 48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931 | head -n 3
1250037181 lr-x------ 1 root root 64 Oct 21 07:00 10000 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)
1250037182 lr-x------ 1 root root 64 Oct 21 07:00 10001 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)
1250037183 lr-x------ 1 root root 64 Oct 21 07:00 10002 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)

Illustration : Total number of kept fd is 11405, number of kept fd for one container is 4882.

[supereagle]

Shell cleanup command:

cd "/proc/`cat /var/run/docker.pid`/fd" | ls -li  | grep '\(deleted\)' | xargs -n 1 -d '\n'  -P 8 -I [] bash -c 'find . -inum `echo "[]" | sed -e "s/^[[:space:]]*//" | cut -d " "  -f 1`  -exec ls -l {} \; -exec  truncate -s 0 {} \;'

This shell command is very slow, it will take 20+ minutes to cleanup 4W files. Following Python script only needs 30 seconds.

Python cleanup script:

#!/usr/bin/env python

# clean deleted file open by docker
import os
import os.path
import sys

pid = open('/var/run/docker.pid', 'r').read()
print 'clean deleted file open by docker: ' + pid
os.chdir('/proc/%s/fd'%pid)

count = 0
for f in os.listdir('.'):
    if not os.path.islink(f):
        continue
    if '(deleted)' in os.readlink(f):
        with open(f, 'w') as of:
            count += 1
            of.truncate(0)
print 'clean %d files'%count