moby / moby

The Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
https://mobyproject.org/
Apache License 2.0
68.71k stars 18.66k forks source link

DISK CRITICAL - /var/lib/docker/devicemapper/mnt/*******2f5388********************* is not accessible: Permission denied #35712

Closed Resvina closed 6 years ago

Resvina commented 6 years ago

Upgrading to docker 17.06.2-ce makes this file "/var/lib/docker/devicemapper/mnt/2f53881f1268a6ec668246f58706f5010b34b52f29656f6fba5caa40d02c6ac5" inaccessible to ec2-user and nagios user even though ec2-user has sudo access and it roke nagios disk check /: $ /usr/lib64/nagios/plugins/check_disk -X nsfs -w 10% -c 5% DISK CRITICAL - /var/lib/docker/devicemapper/mnt/2f53881f1268a6ec668246f58706f5010b34b52f29656f6fba5caa40d02c6ac5 is not accessible: Permission denied

Steps to reproduce the issue:

  1. upgrade to docker 17.06.2-ce 2.from a nagios server, run :/usr/lib64/nagios/plugins/check_disk -vvv -X nsfs -w 10% -c 5%

Describe the results you received: $ /usr/lib64/nagios/plugins/check_disk -vvv -X nsfs -w 10% -c 5% Thresholds(pct) for /proc warn: 10.000000 crit 5.000000 calling stat on /proc Thresholds(pct) for /sys warn: 10.000000 crit 5.000000 calling stat on /sys Thresholds(pct) for / warn: 10.000000 crit 5.000000 calling stat on / For /, used_pct=3 free_pct=97 used_units=3567 free_units=147394 total_units=151059 used_inodes_pct=2 free_inodes_pct=98 fsp.fsu_blocksize=4096 mult=1048576 Freespace_units result=0 Freespace% result=0 Usedspace_units result=0 Usedspace_percent result=0 Usedinodes_percent result=0 Freeinodes_percent result=0 Thresholds(pct) for /dev warn: 10.000000 crit 5.000000 calling stat on /dev For /dev, used_pct=1 free_pct=99 used_units=0 free_units=30695 total_units=30695 used_inodes_pct=1 free_inodes_pct=99 fsp.fsu_blocksize=4096 mult=1048576 Freespace_units result=0 Freespace% result=0 Usedspace_units result=0 Usedspace_percent result=0 Usedinodes_percent result=0 Freeinodes_percent result=0 Thresholds(pct) for /dev/pts warn: 10.000000 crit 5.000000 calling stat on /dev/pts Thresholds(pct) for /dev/shm warn: 10.000000 crit 5.000000 calling stat on /dev/shm For /dev/shm, used_pct=0 free_pct=100 used_units=0 free_units=30705 total_units=30705 used_inodes_pct=1 free_inodes_pct=99 fsp.fsu_blocksize=4096 mult=1048576 Freespace_units result=0 Freespace% result=0 Usedspace_units result=0 Usedspace_percent result=0 Usedinodes_percent result=0 Freeinodes_percent result=0 Thresholds(pct) for /data warn: 10.000000 crit 5.000000 calling stat on /data For /data, used_pct=7 free_pct=93 used_units=12126 free_units=179075 total_units=201458 used_inodes_pct=1 free_inodes_pct=99 fsp.fsu_blocksize=4096 mult=1048576 Freespace_units result=0 Freespace% result=0 Usedspace_units result=0 Usedspace_percent result=0 Usedinodes_percent result=0 Freeinodes_percent result=0 Thresholds(pct) for /proc/sys/fs/binfmt_misc warn: 10.000000 crit 5.000000 calling stat on /proc/sys/fs/binfmt_misc Thresholds(pct) for /cgroup/blkio warn: 10.000000 crit 5.000000 calling stat on /cgroup/blkio Thresholds(pct) for /cgroup/cpu warn: 10.000000 crit 5.000000 calling stat on /cgroup/cpu Thresholds(pct) for /cgroup/cpuacct warn: 10.000000 crit 5.000000 calling stat on /cgroup/cpuacct Thresholds(pct) for /cgroup/cpuset warn: 10.000000 crit 5.000000 calling stat on /cgroup/cpuset Thresholds(pct) for /cgroup/devices warn: 10.000000 crit 5.000000 calling stat on /cgroup/devices Thresholds(pct) for /cgroup/freezer warn: 10.000000 crit 5.000000 calling stat on /cgroup/freezer Thresholds(pct) for /cgroup/hugetlb warn: 10.000000 crit 5.000000 calling stat on /cgroup/hugetlb Thresholds(pct) for /cgroup/memory warn: 10.000000 crit 5.000000 calling stat on /cgroup/memory Thresholds(pct) for /cgroup/perf_event warn: 10.000000 crit 5.000000 calling stat on /cgroup/perf_event Thresholds(pct) for /var/lib/docker/plugins warn: 10.000000 crit 5.000000 calling stat on /var/lib/docker/plugins For /var/lib/docker/plugins, used_pct=3 free_pct=97 used_units=3567 free_units=147394 total_units=151059 used_inodes_pct=2 free_inodes_pct=98 fsp.fsu_blocksize=4096 mult=1048576 Freespace_units result=0 Freespace% result=0 Usedspace_units result=0 Usedspace_percent result=0 Usedinodes_percent result=0 Freeinodes_percent result=0 Thresholds(pct) for /var/lib/docker/devicemapper warn: 10.000000 crit 5.000000 calling stat on /var/lib/docker/devicemapper For /var/lib/docker/devicemapper, used_pct=3 free_pct=97 used_units=3567 free_units=147394 total_units=151059 used_inodes_pct=2 free_inodes_pct=98 fsp.fsu_blocksize=4096 mult=1048576 Freespace_units result=0 Freespace% result=0 Usedspace_units result=0 Usedspace_percent result=0 Usedinodes_percent result=0 Freeinodes_percent result=0 Thresholds(pct) for /var/run/docker/netns/default warn: 10.000000 crit 5.000000 Thresholds(pct) for /var/lib/docker/devicemapper/mnt/2f53881f1268a6ec668246f58706f5010b34b52f29656f6fba5caa40d02c6ac5 warn: 10.000000 crit 5.000000 calling stat on /var/lib/docker/devicemapper/mnt/2f53881f1268a6ec668246f58706f5010b34b52f29656f6fba5caa40d02c6ac5 stat failed on /var/lib/docker/devicemapper/mnt/2f53881f1268a6ec668246f58706f5010b34b52f29656f6fba5caa40d02c6ac5 DISK CRITICAL - /var/lib/docker/devicemapper/mnt/2f53881f1268a6ec668246f58706f5010b34b52f29656f6fba5caa40d02c6ac5 is not accessible: Permission denied

Describe the results you expected: $ /usr/lib64/nagios/plugins/check_disk -X nsfs -w 10% -c 5% DISK OK - free space: / 147266 MB (97% inode=98%); /dev 30696 MB (99% inode=99%); /dev/shm 30704 MB (100% inode=99%); /data 237347 MB (99% inode=99%);| /=3696MB;135953;143506;0;151059 /dev=0MB;27626;29161;0;30696 /dev/shm=0MB;27633;29168;0;30704 /data=1691MB;226669;239262;0;251855

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   3dfb8343b139d6342acfd9975d7f1068b5b1c3d3
 Built:        Tue Oct 17 18:33:39 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   402dd4a/17.06.2-ce
 Built:        Tue Oct 17 18:34:10 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 11
 Running: 4
 Paused: 0
 Stopped: 7
Images: 14
Server Version: 17.06.2-ce
Storage Driver: devicemapper
 Pool Name: docker-docker--pool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: ext4
 Data file:
 Metadata file:
 Data Space Used: 9.082GB
 Data Space Total: 84.95GB
 Data Space Available: 75.86GB
 Metadata Space Used: 2.593MB
 Metadata Space Total: 88.08MB
 Metadata Space Available: 85.49MB
 Thin Pool Minimum Free Space: 8.495GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.41-36.55.amzn1.x86_64
Operating System: Amazon Linux AMI 2016.09
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 59.97GiB
Name: va-ottecs-prd-i-07f39de5fe8cc4601
ID: QG7Z:BJ6H:I4TY:5DLT:MTWO:TOST:EHT3:DVA5:6UEQ:E2XR:MD7U:5TFW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

thaJeztah commented 6 years ago

Those paths are the mount points for the container's filesystem, which likely are not accessible; I'd recommend excluding those paths.

This is a duplicate of https://github.com/moby/moby/issues/27676, so closing this one, but feel free to discuss