Errors with devmapper 'dm_task_run failed'

[josh-shaw-dev]

So I am having issues running the migratory tool, the tool which finishes in about a minute or so, expected to be run four about 6 hours or so with the amount of data.

Trying to upgrade so I can get away from these errors. Referenced here https://github.com/docker/docker/issues/14738 time="2016-03-16T22:40:07.797402232Z" level=error msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container 6d2de143cde8bb3b510c95c44ac115d256909876497d32113ecc6a5d02f73731: adding interface veth2e4eadf to bridge docker0 failed: could not find bridge docker0: no such network interface"

Some more info of use, the /var/lib/docker is linked to a mounted in volume in /var/data on the host

Command run: sudo docker run -d --privileged -v /var/lib/docker:/var/lib/docker docker/v1.10-migrator -s devicemapper

time="2016-03-16T22:43:50Z" level=debug msg="devicemapper: driver version is 4.29.0" 
time="2016-03-16T22:43:50Z" level=warning msg="devmapper: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors. For more information, see https://docs.docker.com/reference/commandline/daemon/#daemon-storage-driver-option" 
time="2016-03-16T22:43:50Z" level=debug msg="devmapper: Generated prefix: docker-202:48-132" 
time="2016-03-16T22:43:50Z" level=debug msg="devmapper: Checking for existence of the pool docker-202:48-132-pool" 
time="2016-03-16T22:43:50Z" level=debug msg="devmapper: poolDataMajMin=7:0 poolMetaMajMin=7:1\n" 
time="2016-03-16T22:43:50Z" level=debug msg="devmapper: Major:Minor for device: /dev/loop0 is:7:0" 
time="2016-03-16T22:43:50Z" level=debug msg="devmapper: Major:Minor for device: /dev/loop1 is:7:1" 
time="2016-03-16T22:43:50Z" level=debug msg="devmapper: loadDeviceFilesOnStart()" 
time="2016-03-16T22:43:50Z" level=debug msg="devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/0022adf8bcd20da4eb6f147ebc48b5c049714ec19d336725d8a0e794d47119ed" 
time="2016-03-16T22:43:50Z" level=debug msg="devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/00fb96427843003189a77a8c1ba52a2c13995588d87f1ae5757d4e1e34a511c3" 
<........More of the same...........>
time="2016-03-16T22:43:51Z" level=debug msg="devmapper: Skipping file /var/lib/docker/devicemapper/metadata/transaction-metadata" 
time="2016-03-16T22:43:51Z" level=debug msg="devmapper: loadDeviceFilesOnStart() END" 
time="2016-03-16T22:43:51Z" level=debug msg="devmapper: constructDeviceIDMap()" 
time="2016-03-16T22:43:51Z" level=debug msg="devmapper: Added deviceId=234 to DeviceIdMap" 
time="2016-03-16T22:43:51Z" level=debug msg="devmapper: Added deviceId=1447 to DeviceIdMap" 
<........More of the same...........>
time="2016-03-16T22:43:51Z" level=debug msg="devmapper: constructDeviceIDMap() END" 
time="2016-03-16T22:43:51Z" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section." 
time="2016-03-16T22:43:51Z" level=debug msg="could not get checksum for \"00fb96427843003189a77a8c1ba52a2c13995588d87f1ae5757d4e1e34a511c3\" with tar-split: \"open /var/lib/docker/graph/00fb96427843003189a77a8c1ba52a2c13995588d87f1ae5757d4e1e34a511c3/tar-data.json.gz: no such file or directory\". Attempting fallback." 
time="2016-03-16T22:43:51Z" level=debug msg="could not get checksum for \"0104771e99512b85255b43fca077ffa3cfc5bc5d6d2f04340d610d027b46661b\" with tar-split: \"open /var/lib/docker/graph/0104771e99512b85255b43fca077ffa3cfc5bc5d6d2f04340d610d027b46661b/tar-data.json.gz: no such file or directory\". Attempting fallback." 
time="2016-03-16T22:43:51Z" level=debug msg="devmapper: activateDeviceIfNeeded(00fb96427843003189a77a8c1ba52a2c13995588d87f1ae5757d4e1e34a511c3)" 
time="2016-03-16T22:43:51Z" level=debug msg="could not get checksum for \"0165379351b4dec84b0d2acbc4705e87f883bfbb91cf641856f2b91323d0015d\" with tar-split: \"open /var/lib/docker/graph/0165379351b4dec84b0d2acbc4705e87f883bfbb91cf641856f2b91323d0015d/tar-data.json.gz: no such file or directory\". Attempting fallback." 
time="2016-03-16T22:43:51Z" level=debug msg="could not get checksum for \"02fa71f1fa3875fbfa00a362387956ee0e7171c7102e9aed7111ad7934296b8b\" with tar-split: \"open /var/lib/docker/graph/02fa71f1fa3875fbfa00a362387956ee0e7171c7102e9aed7111ad7934296b8b/tar-data.json.gz: no such file or directory\". Attempting fallback." 
time="2016-03-16T22:43:51Z" level=error msg="could not calculate checksum for \"00fb96427843003189a77a8c1ba52a2c13995588d87f1ae5757d4e1e34a511c3\", \"Can't create snap device: devmapper: Error activating devmapper device for '00fb96427843003189a77a8c1ba52a2c13995588d87f1ae5757d4e1e34a511c3': devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed\"" 
time="2016-03-16T22:43:51Z" level=debug msg="layer 00fb96427843003189a77a8c1ba52a2c13995588d87f1ae5757d4e1e34a511c3 took 0.01 seconds" 
time="2016-03-16T22:43:51Z" level=debug msg="devmapper: activateDeviceIfNeeded(0104771e99512b85255b43fca077ffa3cfc5bc5d6d2f04340d610d027b46661b)" 
time="2016-03-16T22:43:51Z" level=error msg="could not calculate checksum for \"0104771e99512b85255b43fca077ffa3cfc5bc5d6d2f04340d610d027b46661b\", \"Can't create snap device: devmapper: Error activating devmapper device for '0104771e99512b85255b43fca077ffa3cfc5bc5d6d2f04340d610d027b46661b': devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed\"" 
time="2016-03-16T22:43:51Z" level=debug msg="layer 0104771e99512b85255b43fca077ffa3cfc5bc5d6d2f04340d610d027b46661b took 0.04 seconds" 
<........More of the same...........>

docker info

Containers: 196
Images: 799
Storage Driver: devicemapper
 Pool Name: docker-202:48-132-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 35.1 GB
 Data Space Total: 107.4 GB
 Data Space Available: 15.79 GB
 Metadata Space Used: 55.38 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.092 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/data/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/data/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.20.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 15.26 GiB
Name: *sanitised*

docker version:

Client version: 1.7.1
Client API version: 1.19
Package Version (client): docker-1.7.1-115.el7.x86_64
Go version (client): go1.4.2
Git commit (client): 446ad9b/1.7.1
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Package Version (server): docker-1.7.1-115.el7.x86_64
Go version (server): go1.4.2
Git commit (server): 446ad9b/1.7.1
OS/Arch (server): linux/amd64

[tonistiigi]

Can you check if you get same error on regular upgrade?

[josh-shaw-dev]

I have upgraded to 1.9.1 to get away from the network interface error in the mean time, I cant have it down for the estimated 6 hours or so as its a prod instance.

Ill see if I get some time to clone it and run it locally for more in-depth testing.

[abilash222]

i get the same error in Docker 1.6.2 env

docker run --privileged --rm -v /var/lib/docker:/var/lib/docker docker/v1.10-migrator

time="2016-03-22T08:02:53Z" level=debug msg="layer eff9922ad47c3fa9d350b7959f08f28172624eadec8e62ba1febf8e55ee259d9 took 0.01 seconds" time="2016-03-22T08:02:53Z" level=debug msg="devmapper: activateDeviceIfNeeded(d84fd19c22ca9e54b41d332b01ee3d7067624f2a9271acc707f779d2bdf0ade7)" time="2016-03-22T08:02:53Z" level=error msg="could not calculate checksum for \"d84fd19c22ca9e54b41d332b01ee3d7067624f2a9271acc707f779d2bdf0ade7\", \"Can't create snap device: devmapper: Error activating devmapper device for 'd84fd19c22ca9e54b41d332b01ee3d7067624f2a9271acc707f779d2bdf0ade7': devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed\""

Output of docker version:

Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.4.2 Git commit (client): ba1f6c3/1.6.2 OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.4.2 Git commit (server): ba1f6c3/1.6.2 OS/Arch (server): linux/amd64

Docker Info

Containers: 0 Images: 33 Storage Driver: devicemapper Pool Name: docker-202:2-1705646-pool Pool Blocksize: 65.54 kB Backing Filesystem: extfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 1.412 GB Data Space Total: 107.4 GB Data Space Available: 50.84 GB Metadata Space Used: 2.204 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.145 GB Udev Sync Supported: true Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-12-01) Execution Driver: native-0.2 Kernel Version: 3.10.0-327.10.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) CPUs: 2 Total Memory: 3.451 GiB Name: zdocker-test ID: P36S:DFNJ:VGWR:4P72:5PZS:2NII:FI6D:YSA6:YRFF:BV74:TPYX:P6MD