Open akosyrev opened 5 years ago
Have the same problem. Is there a reason why devicemapper is excluded?
https://github.com/docker/cli/pull/1424 "Deprecate", but it was excluded...
Same here devicemapper is excluded on debian stretch starting from 18.09
ping @andrewhsu
This is being tracked internally; opened an issue for the packaging team.
Same here devicemapper is excluded on debian buster starting from Docker version 18.09.0, build 4d60db4
i got this:
Unable to locate plugin: devicemapper, retrying in 1s
dockerd -s devicemapper --experimental -D
DEBU[2018-11-13T10:38:00.517663555-05:00] Listener created for HTTP on unix (/var/run/docker.sock)
WARN[2018-11-13T10:38:00.521002154-05:00] failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: file exists. Deleting synchronously
DEBU[2018-11-13T10:38:00.521222922-05:00] Golang's threads limit set to 114570
INFO[2018-11-13T10:38:00.521525999-05:00] parsed scheme: "unix" module=grpc
INFO[2018-11-13T10:38:00.521542834-05:00] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2018-11-13T10:38:00.521602212-05:00] parsed scheme: "unix" module=grpc
INFO[2018-11-13T10:38:00.521616361-05:00] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2018-11-13T10:38:00.521683461-05:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}] module=grpc
DEBU[2018-11-13T10:38:00.521716188-05:00] Using default logging driver json-file
DEBU[2018-11-13T10:38:00.521746247-05:00] [graphdriver] trying provided driver: devicemapper
INFO[2018-11-13T10:38:00.521806617-05:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}] module=grpc
INFO[2018-11-13T10:38:00.521846492-05:00] ClientConn switching balancer to "pick_first" module=grpc
INFO[2018-11-13T10:38:00.521727080-05:00] ClientConn switching balancer to "pick_first" module=grpc
WARN[2018-11-13T10:38:00.521924066-05:00] Unable to locate plugin: devicemapper, retrying in 1s
INFO[2018-11-13T10:38:00.521923492-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42019a160, CONNECTING module=grpc
INFO[2018-11-13T10:38:00.521959850-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42016c9e0, CONNECTING module=grpc
INFO[2018-11-13T10:38:00.522111703-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42016c9e0, READY module=grpc
INFO[2018-11-13T10:38:00.522124829-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42019a160, READY module=grpc
DEBU[2018-11-13T10:38:00.522207455-05:00] processing event stream module=libcontainerd namespace=plugins.moby
WARN[2018-11-13T10:38:01.522307695-05:00] Unable to locate plugin: devicemapper, retrying in 2s
WARN[2018-11-13T10:38:03.522773842-05:00] Unable to locate plugin: devicemapper, retrying in 4s
^CINFO[2018-11-13T10:38:04.863216849-05:00] Processing signal 'interrupt'
WARN[2018-11-13T10:38:07.523305063-05:00] Unable to locate plugin: devicemapper, retrying in 8s
^CINFO[2018-11-13T10:38:08.095129525-05:00] Processing signal 'interrupt'
^CINFO[2018-11-13T10:38:08.303084600-05:00] Processing signal 'interrupt'
^CINFO[2018-11-13T10:38:08.558976666-05:00] Processing signal 'interrupt'
INFO[2018-11-13T10:38:08.559126689-05:00] Forcing docker daemon shutdown without cleanup; 3 interrupts received ```
Hi @thaJeztah, what's the status on this issue? Is this resolved now?
@thaJeztah We rely on the block storage provided by devicemapper storage driver, so would really like to see this resolved.
Also, are you planning to support any alternative to devicemapper
? Because block device support provides better performances in our case.
I'm not sure what the current status is; I know this was something that was being looked into, but no details yet if they found a solution.
Also, are you planning to support any alternative to devicemapper?
The current roadmap is to complete the migration to using containerd as a runtime, and for image management (see https://github.com/moby/moby/issues/38043). Currently, the functionality of containerd and dockerd overlaps (as in; they both are capable of image-management, and both implement storage drivers). The goal is to hand over the overlapping parts to containerd. Currently containerd has support for overlay, zfs, btrfs, and aufs as storage drivers ("snapshotters") but additional storage drivers could be added upstream in the containerd project. t.b.h., I'm not sure if devicemapper would find its way in there. Maintenance of devicemapper was largely driven by Red Hat (as it was the only option for their kernels), but now that overlay has become mainstream, focus has shifted in that direction.
Because block device support provides better performances in our case.
Do you have more information about your use-case? Is this for writes or reads ? (for writes, the general recommendation is to use a volume, to skip the CoW performance penalty)
@thaJeztah
I'm not sure if devicemapper would find its way in there. Maintenance of devicemapper was largely driven by Red Hat (as it was the only option for their kernels), but now that overlay has become mainstream, focus has shifted in that direction.
In our case, we get better performance with block device rootfs. This means that we're not strongly attached to devicemapper
itself but to the fact that it exposes block devices. And so it's important for us to keep at least one storage driver that exposes rootfs as block device if we want to maintain this level of performance.
Do you have more information about your use-case?
Because Kata Containers runtime spawn a VM, and because the rootfs needs to be passed inside this VM, we are limited by the device model exposed by QEMU. And in this case, using virtio-block
or virtio-scsi
is more performant than using 9pfs
. But virtio-scsi
is block device based... that's the rationale why we need a storage driver exposing rootfs as block device.
Gotcha, thanks for the extra info. /cc @dmcgowan @crosbymichael
Did anyone get rid from this issue? After upgrading the docker system we had faced the same and downgraded the version. Switch to the overlay2 is the only option?
that is the only option, i rollback to previous version just for export a docker image, them upgrade again and import the docker image...
This is not a "one-stop-shop" solution, but might help anyone looking for migrating to overlay2.
https://github.com/nextcloud/vm/blob/master/static/docker_overlay2.sh
This is a wreckless upgrade, how on earth. I had to downgrade too.
I would pin the package to 18.06 until this issue has been resolved.
cat > /etc/apt/preferences.d/docker <<EOF
Package: docker-ce
Pin: version 18.06*
Pin-Priority: 1000
EOF
apt-get install docker-ce
journalctl -xe | grep docker
I posted this a suggested solution on askubuntu: https://askubuntu.com/a/1111895/24580.
the upgrade's inevitable, the simplest option is a rollback, export, upgrade and import.
What is current status of devicemapper? It is removed in build scripts https://github.com/docker/docker-ce-packaging/blob/master/image/Dockerfile.engine#L43 with this PR https://github.com/docker/docker-ce-packaging/pull/223/files#diff-6670e1ec6985ef6cc31ddd74f1e52e70R43
I cannot find any information about deprecation here https://docs.docker.com/storage/storagedriver/device-mapper-driver/ .
Devicemapper is best choice when you need storage driver operating at the block level (when you need to change 1-2 little parts of big file inside container). I know there are ZFS and BTRFS drivers. I am not sure about their reliability, but I know ZFS does not provide DIRECT_IO for now.
@thaJeztah , do you have any information?
@seemethere do you have more info on the status for devicemapper in the .deb packages?
still looking for ETA on fix -- we have plenty of developers who are looking to test Kata with Firecracker, and I am uncomfortable suggesting an old version of Docker who's support window is winding down.
Is there an ETA?
@thaJeztah is this an issue only for .deb packages? In any case, it seems that someone with this problem (.deb or all) needs to upgrade to 18.09.2 (to address the latest CVE) and switch to overlay2 at once.
This is only an issue with .deb
packages of 18.09.x. To address the CVE, there's also an updated 18.06 package (18.06.2), which contains a patched version of runc to mitigate the CVE, and should still have support for device mapper in the .deb
packages. One restriction for that package, is that the fix for the CVE does not work on Ubuntu 14.04 when running the original 3.13 kernel; before updating the package, upgrade to an Ubuntu provided 4.x kernel for Ubuntu 14.04.
Is this feature gone from 18.09? While engine lacks a feature-deprecation-policy on this page, it would be nice if documentation and warnings happened.
https://docs.docker.com/engine/deprecated/#device-mapper-storage-driver
For users who use LVM with low latency NVMe drives with some test cases like DBs Overlay2 is a huge and breaking change. I am extremely disappointed that there is no note that this feature is removed without notification. I get that loopback-lvm performed pretty poorly but it would be nice if depreciation vs removal was accurately documented.
²) The devicemapper storage driver is deprecated in Docker Engine 18.09, and will be removed in a future release. It is recommended that users of the devicemapper storage driver migrate to overlay2.
It would be nice to know if I have to find a new backing storage solution for some needs and if this is marked as a removed feature without a deprecation period if that is has been decided.
I can't get a straight answer on this from Docker-CE package maintainers (all at Docker Inc), it's quite frustrating.
@thaJeztah Can someone just remove that tag 'exclude_graphdriver_devicemapper' ?
I think this issue was related to changes in packaging, to facilitate upgrading/updating the engine versions through images (docker engine activate
, docker engine update
); (I think) removing that tag caused builds to fail, but I don't know if a solution for that was found already.
So, break existing users for a feature that nobody is using? 👻
FWIW, it's still in the .rpm
packages (CentOS, RHEL, Fedora), but not in the .deb
(Debian, Ubuntu)
That just makes no sense to me. Are these new features not supported on CentOS/Fedora/RHEL?
This is a pretty clear cut break with no explanation at all, not even "oh we messed up but aren't going to fix it either", just silence.
Docker CE 18.09.x works with CentOS and Fedora configured with devicemapper. Docker does not qualify Docker CE on RHEL before release, but it is likely to work based on how RHEL is related to CentOS.
We intended to have Docker CE 18.09.x work with Ubuntu and Debian configured with devicemapper as well, but we have encountered complications with the way we build the binaries for the packages that was discovered after release. This is because we did not qualify Docker CE 18.09.x on devicemapper, but on the newer-supported overlay2 which we intend to have as the default, hence the deprecation notice for devicemapper.
Apologies for the problem this has caused existing devicemapper users on Ubuntu and Debian. This is not the way we intend to phase out support for something we know as common use. This is a bug that has definitely slipped through the cracks after the initial 18.09.0 release.
I'll circle back around with the team on what we gotta do to get the devicemapper support back in for 18.09.x on Ubuntu and Debian. In the meantime, if migration is an option, there are instructions for howto convert devicemapper to overlay2, to be performed on your system before upgrade to 18.09.x obviously.
Thanks for taking a look, @andrewhsu.
@andrewhsu
I hope you realize that there are a lot of layman users like me out there, I just use Docker when the app developers recommend as a preferred method. I do not do Docker dev or Linux system configurations. I am just a basic user who needs to use Docker because it is recommended. When the system breaks like this in the middle of nothing, it is very hard to make sense of what is going on in these situations. I did not know that my Docker appps all were down, until when I realized that all my web apps were down.
If the migration is the only option then please provide a way to automatically migrate all this stuff.
@andrewhsu there are reasons as a non-layman user I explicitly CHOSE to use devicemapper rather than overlay2. None of the documentation I've seen has stated WHY overlay2 is better than devicemapper, or refutes the reasons that I and many thousands of others chose devicemapper.
My reasons? Once configured properly (which isn't that hard, and is easily automated with ansible), the devicemapper thinpool provides what I believe to be the fastest-possible performance. And it enables automatic LVM volume-size extension; I can allocate (say) 10GB on a terabyte physical volume in set-and-forget fashion. The volume might grow to 50GB or 100GB but I can still use the rest for purposes other than docker images.
In a nutshell: I went through the list of available storage drivers circa Feb 2017, chose devicemapper, and have had no reason to reconsider that choice--until the snafu with your company's CI failure in the 18.09 release cycle. And now I have to assume that your managers are busily reprioritizing engineering tasks and aren't likely to ever revisit devicemapper. In any event, before closing this ticket out as won't-fix, please have someone document the advantages of overlay2 better and explain the disadvantages of devicemapper (if any). Because I still don't see why I'm forced to make this change, and I'm concerned about suddenly losing autoextend or I/O performance afterward.
In the meantime, if migration is an option, there are instructions for howto convert devicemapper to overlay2, to be performed on your system before upgrade to 18.09.x obviously.
The referenced procedure doesn't lay out how to migrate an existing devicemapper install. It basically just tells you to scratch your existing storage and create a new one, which I dont think is an acceptable migration procedure in any case.
Also an issue on Xenial.
With docker 18.09 using containerd as the runtime handler, is there a way to docker use a storage-driver that exists in containerd? There are multiple efforts to add block device support to containerd here, here and here. It will be nice to allow one of these to be the default storage driver for docker.
With docker 18.09 using containerd as the runtime handler, is there a way to docker use a storage-driver that exists in containerd?
Not possible yet, as those parts are not yet integrated with the docker daemon (storage-drivers are still handled by dockerd). Work is in progress to integrate those parts; see https://github.com/moby/moby/pull/38738 and https://github.com/moby/moby/pull/38925)
Hey guys!
The latest nightly builds after https://github.com/docker/docker-ce-packaging/pull/314 was merged should contain devicemapper
support on Ubuntu and Debian and should be included in the next release for Docker CE!
You can install the latest nightly builds with get.docker.com using:
curl -fsSL get.docker.com | CHANNEL=nightly sh
Ran a quick test with Ubuntu 18.04 to verify:
~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
~$ docker --version
Docker version 0.0.0-20190329010133-83264de7d5, build 83264de7d5
~$ docker info 2>/dev/null | grep Storage
Storage Driver: devicemapper
Keep in mind that devicemapper
will still be deprecated at some point in the future so moving to overlay2
is still preferred.
@seemethere - can the deprecation be held off until https://github.com/docker/for-linux/issues/452#issuecomment-477538868 is addressed?
For some users, overlay2 isn't an option. Once above is addressed, they'll have a viable and supported option.
@thaJeztah WDYT?
Devicemapper will still be there in the upcoming (19.03) release. In addition, work is in progress in containerd to provide block-device snapshotters;
Yep - I just want to make sure containerd snapshotters work with docker (see Ganesh’s prior comment).
Even though https://github.com/docker/docker-ce-packaging/pull/314 has been released, this issue is still present for 18.09.5 for yum repositories
I don't understand how a company with as reputable a name as Docker allows an issue this disruptive, first, to happen, or worse, to continue effectively unaddressed and untreated for over six months, despite the caliber of professionals which continue to (for all intents and purposes) politely petition an appropriate and open-minded response or set of solutions.
Attempting to use the "supported" and functional ZFS driver, which unfortunately no longer works due to this change.
Use Overlay2? No! And my reasons are just as valid as the infinitely diverse other reasons had and given by other sysadmins here and elsewhere.
to continue effectively unaddressed and untreated for over six months
Not sure if you read the prior comments, this was addressed and taken seriously. If you are continuing to have a problem after the proposed solution, please highlight what steps you tried and how they have failed.
Use Overlay2? No!
No one is forcing anyone to use overlay2. The devicemapper driver in Docker is being deprecated due to lack of maintenance, it is not being immediately removed. Rather devicemapper support will be done through containerd in the future and in the long term replaces the deprecated graphdriver implementation. I think this wasn't communicated well and the unintentional package breakage gave the impression that they could not longer use devicemapper with Docker anymore. That is not the case, please see the work being done in containerd to support for block device snapshotters.
@thaJeztah @seemethere if this issue has been fixed can you close this issue?
@dmcgowan as noted above already, the issue hasn't been fixed in a way that we users have access to. Until a 19.x version comes out, we're having to run back-ported updates of 18.06.x.
This issue has been resolved in 19.03.X, will close this issue once a GA version of that package is available.
This issue has been resolved in 19.03.X, will close this issue once a GA version of that package is available.
https://github.com/docker/docker-ce/releases/tag/v19.03.1 19.03 has been released. Should we close this issue?
I found that last nightly build from this repo https://download.docker.com/linux/ubuntu/dists/xenial/pool/nightly/amd64/ fails with error, when i'm trying to run it with devicemapper:
Error message: