search

elastio / elastio-snap

kernel module for taking block-level snapshots and incremental backups of Linux block devices
GNU General Public License v2.0
19 stars 7 forks source link

Pulling docker image will cause an umount snapshot error in root partition #294

Open jamesruic opened 1 year ago

jamesruic commented 1 year ago

I use the latest driver in Ubuntu20.04. It will cause a umount error, when I run a busybox container. Does elastio support docker or something wrong with my steps?

My machine info:

root@ubuntu20:~# uname -r
5.4.0-131-generic
root@ubuntu20:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
root@ubuntu20:~# lsblk -f
NAME          FSTYPE   LABEL     UUID                                 FSAVAIL FSUSE% MOUNTPOINT
elastio-snap0
vda
├─vda1        vfat               A11B-DE03                              93.3M     5% /boot/efi
├─vda2        ext4               6f98bc46-e03c-47ae-b95a-d45eae3ad2cd  382.3M    52% /
└─vda3        swap               8c9c3869-6be8-40ae-9f56-104df5e06ecf                [SWAP]

  1. create snapshot success:

    root@ubuntu20:~# cat /proc/elastio-snap-info
{
    "version": "0.12.2",
    "devices": [
            {
                    "minor": 0,
                    "cow_file": "/.snapshot0",
                    "block_device": "/dev/vda2",
                    "max_cache": 314572800,
                    "fallocate": 1610612736,
                    "seq_id": 1,
                    "uuid": "1b26b15fb537402ab666d75a9b7c73ec",
                    "version": 1,
                    "nr_changed_blocks": 1,
                    "state": 3,
                    "ignore_errors": 0,
                    "cow_on_bdev": true
            }
    ]
}

  2. get the latest busybox image and run

    docker pull busybox:latest
docker run -it busybox

  3. got error state -5

    root@ubuntu20:~# cat /proc/elastio-snap-info
{
    "version": "0.12.2",
    "devices": [
            {
                    "minor": 0,
                    "cow_file": "/.snapshot0",
                    "block_device": "/dev/vda2",
                    "max_cache": 314572800,
                    "fallocate": 1610612736,
                    "seq_id": 1,
                    "uuid": "1b26b15fb537402ab666d75a9b7c73ec",
                    "version": 1,
                    "nr_changed_blocks": 580,
                    "error": -5,
                    "state": 1,
                    "ignore_errors": 0,
                    "cow_on_bdev": true
            }
    ]
}

    dmesg:

    [11000.353369] elastio-snap: ioctl command received: 1076379905
[11000.353383] elastio-snap: received setup snap ioctl - 0 : /dev/vda2 : /.snapshot0 : no_ignore_errors
[11000.353407] elastio-snap: allocating device struct
[11000.353409] elastio-snap: initializing tracer
[11000.353411] elastio-snap: finding block device
[11000.353415] elastio-snap: checking block device is not already being traced
[11000.353416] elastio-snap: fetching the absolute pathname for the base device
[11000.353425] elastio-snap: calculating block device size and offset
[11000.353428] elastio-snap: bdev size = 31457280, offset = 206848
[11000.353432] elastio-snap: creating cow manager
[11000.353434] elastio-snap: allocating cow manager, seqid = 1
[11000.353436] elastio-snap: creating cow file
[11000.362585] elastio-snap: allocating cow manager array (960 sections)
[11000.375839] elastio-snap: writing cow header
[11000.375875] elastio-snap: finding cow file inode
[11000.375877] elastio-snap: getting relative pathname of cow file
[11000.376518] elastio-snap: allocating queue and setting up make request function
[11000.376592] elastio-snap: allocating gendisk
[11000.376599] elastio-snap: setting up make request function
[11000.376601] elastio-snap: setting queue limits
[11000.376604] elastio-snap: initializing gendisk
[11000.376605] elastio-snap: naming gendisk
[11000.376609] elastio-snap: block device size: 31457280
[11000.376613] elastio-snap: adding disk
[11000.377132] elastio-snap: starting mrf kernel thread
[11000.377255] elastio-snap: creating kernel cow thread
[11000.377329] elastio-snap: getting the base block device's make_request_fn
[11000.377337] elastio-snap: freezing 'vda2'
[11000.428495] elastio-snap: detected block device mount: none -> / : 0x44000
[11000.428510] elastio-snap: mount returned: 0
[11000.455616] elastio-snap: starting tracing
[11000.455618] elastio-snap: thawing 'vda2'
[11000.471396] elastio-snap: minor range = 0 - 0
[11314.224167] elastio-snap: detected block device mount: overlay -> /var/lib/docker/overlay2/3d67a6e217375618d04c9d2192c772b268ef6e7176a4f5d50ae7d10352f67391-init/merged : 0x0
[11314.224366] elastio-snap: mount returned: 0
[11314.225161] elastio-snap: detected block device umount: /var/lib/docker/overlay2/3d67a6e217375618d04c9d2192c772b268ef6e7176a4f5d50ae7d10352f67391-init/merged : 2
[11314.225201] elastio-snap: umount returned: 0
[11314.263458] elastio-snap: detected block device mount: overlay -> /var/lib/docker/overlay2/3d67a6e217375618d04c9d2192c772b268ef6e7176a4f5d50ae7d10352f67391/merged : 0x0
[11314.271297] elastio-snap: mount returned: 0
[11314.271427] elastio-snap: detected block device umount: /var/lib/docker/overlay2/3d67a6e217375618d04c9d2192c772b268ef6e7176a4f5d50ae7d10352f67391/merged : 2
[11314.277676] elastio-snap: umount returned: 0
[11314.316459] elastio-snap: detected block device mount: overlay -> /var/lib/docker/overlay2/3d67a6e217375618d04c9d2192c772b268ef6e7176a4f5d50ae7d10352f67391/merged : 0x0
[11314.317447] elastio-snap: mount returned: 0
[11314.339729] docker0: port 1(vethb0e4b9a) entered blocking state
[11314.339731] docker0: port 1(vethb0e4b9a) entered disabled state
[11314.339930] device vethb0e4b9a entered promiscuous mode
[11314.403623] elastio-snap: detected block device mount: /proc/self/exe -> /var/run/docker/runtime-runc/moby/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e/runc.bOcIdZ : 0x1000
[11314.403733] elastio-snap: mount returned: 0
[11314.403735] elastio-snap: detected block device mount:  -> /var/run/docker/runtime-runc/moby/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e/runc.bOcIdZ : 0x1021
[11314.403799] elastio-snap: mount returned: 0
[11314.403803] elastio-snap: detected block device umount: /var/run/docker/runtime-runc/moby/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e/runc.bOcIdZ : 2
[11314.403805] elastio-snap: block device umount detected for device 0
[11314.403805] elastio-snap: stopping cow thread
[11314.403815] elastio-snap: snap_cow_thread() done.
[11314.403823] elastio-snap: destroying cow manager. close method: 2
[11314.406495] elastio-snap: getting cow file extents from filp=000000004a47e0ec
[11314.406496] elastio-snap: attempting page stealing from exe
[11314.406520] elastio-snap: fiemap for cow file (ret 0), extents 37 (max 1024)
[11314.406521] elastio-snap:    cow file extent: log 0x0, phy 0x2ce800000, len 4096
[11314.406522] elastio-snap:    cow file extent: log 0x1000, phy 0x2ce801000, len 31457280
[11314.406522] elastio-snap:    cow file extent: log 0x1e01000, phy 0x2d0601000, len 1859584
[11314.406523] elastio-snap:    cow file extent: log 0x1fc7000, phy 0x2d07c7000, len 92508160
[11314.406523] elastio-snap:    cow file extent: log 0x7800000, phy 0x2d6000000, len 125829120
[11314.406523] elastio-snap:    cow file extent: log 0xf000000, phy 0x2dd800000, len 125829120
[11314.406524] elastio-snap:    cow file extent: log 0x16800000, phy 0x2e5000000, len 125829120
[11314.406524] elastio-snap:    cow file extent: log 0x1e000000, phy 0x2ec800000, len 125829120
[11314.406524] elastio-snap:    cow file extent: log 0x25800000, phy 0x2f4000000, len 125829120
[11314.406525] elastio-snap:    cow file extent: log 0x2d000000, phy 0x2fb800000, len 75497472
[11314.406525] elastio-snap:    cow file extent: log 0x31800000, phy 0x319800000, len 125829120
[11314.406525] elastio-snap:    cow file extent: log 0x39000000, phy 0x321000000, len 125829120
[11314.406526] elastio-snap:    cow file extent: log 0x40800000, phy 0x328800000, len 25165824
[11314.406526] elastio-snap:    cow file extent: log 0x42000000, phy 0x32b000000, len 83886080
[11314.406527] elastio-snap:    cow file extent: log 0x47000000, phy 0x32a001000, len 8384512
[11314.406527] elastio-snap:    cow file extent: log 0x477ff000, phy 0x332d67000, len 4096
[11314.406527] elastio-snap:    cow file extent: log 0x47800000, phy 0x2c8206000, len 6266880
[11314.406528] elastio-snap:    cow file extent: log 0x47dfa000, phy 0x332d68000, len 2121728
[11314.406528] elastio-snap:    cow file extent: log 0x48000000, phy 0x2b7e1b000, len 1986560
[11314.406528] elastio-snap:    cow file extent: log 0x481e5000, phy 0x2bfc00000, len 1249280
[11314.406529] elastio-snap:    cow file extent: log 0x48316000, phy 0x2c80fb000, len 1081344
[11314.406529] elastio-snap:    cow file extent: log 0x4841e000, phy 0x2b7d00000, len 1048576
[11314.406532] elastio-snap:    cow file extent: log 0x4851e000, phy 0x332f6e000, len 598016
[11314.406532] elastio-snap:    cow file extent: log 0x485b0000, phy 0x2c806b000, len 40960
[11314.406532] elastio-snap:    cow file extent: log 0x485ba000, phy 0x2bf800000, len 20480
[11314.406533] elastio-snap:    cow file extent: log 0x485bf000, phy 0x2bf807000, len 12288
[11314.406533] elastio-snap:    cow file extent: log 0x485c2000, phy 0x2bfa01000, len 2093056
[11314.406534] elastio-snap:    cow file extent: log 0x487c1000, phy 0x2bf80c000, len 2048000
[11314.406534] elastio-snap:    cow file extent: log 0x489b5000, phy 0x2c80a5000, len 352256
[11314.406534] elastio-snap:    cow file extent: log 0x48a0b000, phy 0x2c8032000, len 233472
[11314.406535] elastio-snap:    cow file extent: log 0x48a44000, phy 0x2ce000000, len 6012928
[11314.406535] elastio-snap:    cow file extent: log 0x49000000, phy 0x2c9000000, len 83886080
[11314.406535] elastio-snap:    cow file extent: log 0x4e000000, phy 0x331800000, len 16777216
[11314.406536] elastio-snap:    cow file extent: log 0x4f000000, phy 0x330000000, len 16777216
[11314.406536] elastio-snap:    cow file extent: log 0x50000000, phy 0x333000000, len 125829120
[11314.406536] elastio-snap:    cow file extent: log 0x57800000, phy 0x33a800000, len 125829120
[11314.406537] elastio-snap:    cow file extent: log 0x5f000000, phy 0x342000000, len 16777216
[11314.406550] elastio-snap: preserving cow file extents
[11314.406709] elastio-snap: device still mounted after umounting cow file's file-system. entering error state: -5
[11314.407818] elastio-snap: umount returned: 0
[11314.445630] elastio-snap: detected block device mount:  -> / : 0x84000
[11314.445660] elastio-snap: mount returned: 0
[11314.445813] elastio-snap: detected block device mount: /var/lib/docker/overlay2/3d67a6e217375618d04c9d2192c772b268ef6e7176a4f5d50ae7d10352f67391/merged -> /var/lib/docker/overlay2/3d67a6e217375618d04c9d2192c772b268ef6e7176a4f5d50ae7d10352f67391/merged : 0x5000
[11314.445820] elastio-snap: mount returned: 0
[11314.445864] elastio-snap: detected block device mount: proc -> /proc/self/fd/7 : 0xe
[11314.445875] elastio-snap: mount returned: 0
[11314.445904] elastio-snap: detected block device mount: tmpfs -> /proc/self/fd/7 : 0x1000002
[11314.445978] elastio-snap: mount returned: 0
[11314.446041] elastio-snap: detected block device mount: devpts -> /proc/self/fd/7 : 0xa
[11314.446123] elastio-snap: mount returned: 0
[11314.446157] elastio-snap: detected block device mount: sysfs -> /proc/self/fd/7 : 0xf
[11314.446180] elastio-snap: mount returned: 0
[11314.446393] elastio-snap: detected block device mount: tmpfs -> /proc/self/fd/7 : 0xe
[11314.446469] elastio-snap: mount returned: 0
[11314.446533] elastio-snap: detected block device mount: /sys/fs/cgroup/systemd/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.446537] elastio-snap: mount returned: 0
[11314.446606] elastio-snap: detected block device mount: /sys/fs/cgroup/systemd/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.446608] elastio-snap: mount returned: 0
[11314.446691] elastio-snap: detected block device mount: /sys/fs/cgroup/cpu,cpuacct/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.446695] elastio-snap: mount returned: 0
[11314.446732] elastio-snap: detected block device mount: /sys/fs/cgroup/cpu,cpuacct/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.446734] elastio-snap: mount returned: 0
[11314.446792] elastio-snap: detected block device mount: /sys/fs/cgroup/devices/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.446797] elastio-snap: mount returned: 0
[11314.446839] elastio-snap: detected block device mount: /sys/fs/cgroup/devices/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.446841] elastio-snap: mount returned: 0
[11314.446883] elastio-snap: detected block device mount: /sys/fs/cgroup/hugetlb/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.446887] elastio-snap: mount returned: 0
[11314.446920] elastio-snap: detected block device mount: /sys/fs/cgroup/hugetlb/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.446922] elastio-snap: mount returned: 0
[11314.446965] elastio-snap: detected block device mount: /sys/fs/cgroup/rdma -> /proc/self/fd/7 : 0x500f
[11314.446968] elastio-snap: mount returned: 0
[11314.446995] elastio-snap: detected block device mount: /sys/fs/cgroup/rdma -> /proc/self/fd/7 : 0x502f
[11314.446997] elastio-snap: mount returned: 0
[11314.447044] elastio-snap: detected block device mount: /sys/fs/cgroup/cpuset/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.447048] elastio-snap: mount returned: 0
[11314.447078] elastio-snap: detected block device mount: /sys/fs/cgroup/cpuset/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.447080] elastio-snap: mount returned: 0
[11314.447119] elastio-snap: detected block device mount: /sys/fs/cgroup/blkio/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.447123] elastio-snap: mount returned: 0
[11314.447149] elastio-snap: detected block device mount: /sys/fs/cgroup/blkio/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.447151] elastio-snap: mount returned: 0
[11314.447200] elastio-snap: detected block device mount: /sys/fs/cgroup/pids/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.447204] elastio-snap: mount returned: 0
[11314.447234] elastio-snap: detected block device mount: /sys/fs/cgroup/pids/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.447236] elastio-snap: mount returned: 0
[11314.447275] elastio-snap: detected block device mount: /sys/fs/cgroup/freezer/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.447279] elastio-snap: mount returned: 0
[11314.447310] elastio-snap: detected block device mount: /sys/fs/cgroup/freezer/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.447312] elastio-snap: mount returned: 0
[11314.447354] elastio-snap: detected block device mount: /sys/fs/cgroup/net_cls,net_prio/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.447358] elastio-snap: mount returned: 0
[11314.447396] elastio-snap: detected block device mount: /sys/fs/cgroup/net_cls,net_prio/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.447398] elastio-snap: mount returned: 0
[11314.447441] elastio-snap: detected block device mount: /sys/fs/cgroup/memory/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.447445] elastio-snap: mount returned: 0
[11314.447476] elastio-snap: detected block device mount: /sys/fs/cgroup/memory/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.447477] elastio-snap: mount returned: 0
[11314.447519] elastio-snap: detected block device mount: /sys/fs/cgroup/perf_event/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x500f
[11314.447523] elastio-snap: mount returned: 0
[11314.447550] elastio-snap: detected block device mount: /sys/fs/cgroup/perf_event/docker/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e -> /proc/self/fd/7 : 0x502f
[11314.447552] elastio-snap: mount returned: 0
[11314.447608] elastio-snap: detected block device mount: mqueue -> /proc/self/fd/7 : 0xe
[11314.447615] elastio-snap: mount returned: 0
[11314.447646] elastio-snap: detected block device mount: shm -> /proc/self/fd/7 : 0xe
[11314.447718] elastio-snap: mount returned: 0
[11314.447757] elastio-snap: detected block device mount: /var/lib/docker/containers/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e/resolv.conf -> /proc/self/fd/7 : 0x5000
[11314.447761] elastio-snap: mount returned: 0
[11314.447772] elastio-snap: detected block device mount:  -> /proc/self/fd/7 : 0x44000
[11314.447774] elastio-snap: mount returned: 0
[11314.447808] elastio-snap: detected block device mount: /var/lib/docker/containers/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e/hostname -> /proc/self/fd/7 : 0x5000
[11314.447813] elastio-snap: mount returned: 0
[11314.447824] elastio-snap: detected block device mount:  -> /proc/self/fd/7 : 0x44000
[11314.447825] elastio-snap: mount returned: 0
[11314.447853] elastio-snap: detected block device mount: /var/lib/docker/containers/70e479f364fad7367dcc8f4ea61d2c8747f6352fd73c4fd7d2896f5d7c271c6e/hosts -> /proc/self/fd/7 : 0x5000
[11314.447857] elastio-snap: mount returned: 0
[11314.447867] elastio-snap: detected block device mount:  -> /proc/self/fd/7 : 0x44000
[11314.447869] elastio-snap: mount returned: 0
[11314.485811] elastio-snap: detected block device mount: /proc/84916/ns/net -> /var/run/docker/netns/fff81f9fbd15 : 0x1000
[11314.485879] elastio-snap: mount returned: 0
skypodolsky commented 1 year ago

@jamesruic thanks for the finding. I think this might be something we already track in #292. Can you please clarify if this happens just after the docker run command, or have you performed some additional actions between the setup-snapshot and docker run? It seems like the driver is trying to unmount the root volume, but the reason for this is uncertain to me. I will take a closer look at that one.

jamesruic commented 1 year ago

@skypodolsky thank you for your reply. I do nothing between setup-snapshot and docker run. After this issue, I made some adjustments in umount_hook for workaround. If block device is a docker service just run orig_umount and return, and it works fine. It is confuse why docker run will trigger umount_hook for me.

jamesruic commented 1 year ago

I found an issue in docker that might be relevant to this situation. https://github.com/moby/moby/issues/36149

skypodolsky commented 1 year ago

@jamesruic there were a couple of commits that might be fixing your problem. If you still have an opportunity, could you please try the latest master?

jamesruic commented 1 year ago

Hi @skypodolsky, I try the latest master version of elastio-snap. I got the same error state when I pulling a docker image. However, when I create the snapshot after pulling the docker image. My container and elastio-snap works good.

dmesg when pulling docker image:

[  906.363420] elastio-snap: creating cow file
[  906.364364] elastio-snap: allocating cow manager array (864 sections)
[  906.365820] elastio-snap: writing cow header
[  906.365829] elastio-snap: finding cow file inode
[  906.365830] elastio-snap: getting relative pathname of cow file
[  906.366022] elastio-snap: allocating queue and setting up make request function
[  906.366035] elastio-snap: allocating gendisk
[  906.366037] elastio-snap: setting up make request function
[  906.366038] elastio-snap: setting queue limits
[  906.366042] elastio-snap: initializing gendisk
[  906.366042] elastio-snap: naming gendisk
[  906.366043] elastio-snap: block device size: 28311552
[  906.366045] elastio-snap: adding disk
[  906.366138] elastio-snap: starting mrf kernel thread
[  906.369349] elastio-snap: creating kernel cow thread
[  906.370616] elastio-snap: getting the base block device's make_request_fn
[  906.370618] elastio-snap: freezing 'dm-0'
[  906.376393] elastio-snap: starting tracing
[  906.376393] elastio-snap: thawing 'dm-0'
[  906.382701] elastio-snap: detected block device mount: none -> / : 0x44000
[  906.382705] elastio-snap: mount returned: 0
[  906.422443] elastio-snap: minor range = 0 - 0
[  929.985389] elastio-snap: detected block device mount:  -> / : 0x84000
[  929.985393] elastio-snap: mount returned: 0
[  929.985539] elastio-snap: detected block device mount: /var/lib/docker/overlay2/754d247bba1e5b883d7053003081ce406e08648182bd53317c326ba80dd39559/diff -> /var/lib/docker/overlay2/754d247bba1e5b883d7053003081ce406e08648182bd53317c326ba80dd39559/diff : 0x5000
[  929.985546] elastio-snap: mount returned: 0
[  929.985630] elastio-snap: detected block device mount:  -> /.pivot_root4171602565 : 0x44000
[  929.985632] elastio-snap: mount returned: 0
[  929.985634] elastio-snap: detected block device umount: /.pivot_root4171602565 : 2
[  929.985635] elastio-snap: block device umount detected for device 0
[  929.985636] elastio-snap: stopping cow thread
[  929.985640] elastio-snap: snap_cow_thread() done.
[  929.985661] elastio-snap: destroying cow manager. close method: 2
[  929.988941] elastio-snap: getting cow file extents from filp=00000000e0a92610
[  929.988942] elastio-snap: attempting page stealing from dockerd
[  929.988956] elastio-snap: fiemap for cow file (ret 0), extents 14 (max 1024)
[  929.988957] elastio-snap:    cow file extent: log 0x0, phy 0x1d8800000, len 4096
[  929.988957] elastio-snap:    cow file extent: log 0x1000, phy 0x1d8801000, len 125825024
[  929.988958] elastio-snap:    cow file extent: log 0x7800000, phy 0x1e0000000, len 125829120
[  929.988958] elastio-snap:    cow file extent: log 0xf000000, phy 0x1e7800000, len 125829120
[  929.988958] elastio-snap:    cow file extent: log 0x16800000, phy 0x1ef000000, len 125829120
[  929.988959] elastio-snap:    cow file extent: log 0x1e000000, phy 0x1f6800000, len 125829120
[  929.988959] elastio-snap:    cow file extent: log 0x25800000, phy 0x1fe000000, len 33554432
[  929.988959] elastio-snap:    cow file extent: log 0x27800000, phy 0x216000000, len 33554432
[  929.988960] elastio-snap:    cow file extent: log 0x29800000, phy 0x21a000000, len 125829120
[  929.988960] elastio-snap:    cow file extent: log 0x31000000, phy 0x221800000, len 125829120
[  929.988960] elastio-snap:    cow file extent: log 0x38800000, phy 0x229000000, len 125829120
[  929.988961] elastio-snap:    cow file extent: log 0x40000000, phy 0x230800000, len 125829120
[  929.988961] elastio-snap:    cow file extent: log 0x47800000, phy 0x238000000, len 125829120
[  929.988961] elastio-snap:    cow file extent: log 0x4f000000, phy 0x23f800000, len 124153856
[  929.988972] elastio-snap: preserving cow file extents
[  929.988992] elastio-snap: device still mounted after umounting cow file's file-system. entering error state: -5
[  929.989020] elastio-snap: umount returned: 0