search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
751 stars 84 forks source link

no devicemapper support in 18.09.x on ubuntu and debian #452

Open akosyrev opened 5 years ago

akosyrev commented 5 years ago

I found that last nightly build from this repo https://download.docker.com/linux/ubuntu/dists/xenial/pool/nightly/amd64/ fails with error, when i'm trying to run it with devicemapper:

{
    "storage-driver": "devicemapper"
}

Error message: 


failed to start daemon: error initializing graphdriver: driver not supported```

I recompiled it from sources and error is still here. iI built from commit 7718f802c486607f080ab8158953dc668e3f09fb

Also I noticed that bightly build from 2018-09-26 is less than 2018-09-20: 17M vs 24M
akosyrev commented 5 years ago

https://github.com/docker/docker-ce-packaging/pull/223/files#diff-6670e1ec6985ef6cc31ddd74f1e52e70R43

domste commented 5 years ago

Have the same problem. Is there a reason why devicemapper is excluded?

Halo9Pan commented 5 years ago

https://github.com/docker/cli/pull/1424 "Deprecate", but it was excluded...

teadur commented 5 years ago

Same here devicemapper is excluded on debian stretch starting from 18.09

cpuguy83 commented 5 years ago

ping @andrewhsu

thaJeztah commented 5 years ago

This is being tracked internally; opened an issue for the packaging team.

nemesis545 commented 5 years ago

Same here devicemapper is excluded on debian buster starting from Docker version 18.09.0, build 4d60db4

nemesis545 commented 5 years ago

i got this:
Unable to locate plugin: devicemapper, retrying in 1s dockerd -s devicemapper --experimental -D 


DEBU[2018-11-13T10:38:00.517663555-05:00] Listener created for HTTP on unix (/var/run/docker.sock) 
WARN[2018-11-13T10:38:00.521002154-05:00] failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: file exists. Deleting synchronously 
DEBU[2018-11-13T10:38:00.521222922-05:00] Golang's threads limit set to 114570         
INFO[2018-11-13T10:38:00.521525999-05:00] parsed scheme: "unix"                         module=grpc
INFO[2018-11-13T10:38:00.521542834-05:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-11-13T10:38:00.521602212-05:00] parsed scheme: "unix"                         module=grpc
INFO[2018-11-13T10:38:00.521616361-05:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-11-13T10:38:00.521683461-05:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
DEBU[2018-11-13T10:38:00.521716188-05:00] Using default logging driver json-file       
DEBU[2018-11-13T10:38:00.521746247-05:00] [graphdriver] trying provided driver: devicemapper 
INFO[2018-11-13T10:38:00.521806617-05:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2018-11-13T10:38:00.521846492-05:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2018-11-13T10:38:00.521727080-05:00] ClientConn switching balancer to "pick_first"  module=grpc
WARN[2018-11-13T10:38:00.521924066-05:00] Unable to locate plugin: devicemapper, retrying in 1s 
INFO[2018-11-13T10:38:00.521923492-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42019a160, CONNECTING  module=grpc
INFO[2018-11-13T10:38:00.521959850-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42016c9e0, CONNECTING  module=grpc
INFO[2018-11-13T10:38:00.522111703-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42016c9e0, READY  module=grpc
INFO[2018-11-13T10:38:00.522124829-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42019a160, READY  module=grpc
DEBU[2018-11-13T10:38:00.522207455-05:00] processing event stream                       module=libcontainerd namespace=plugins.moby
WARN[2018-11-13T10:38:01.522307695-05:00] Unable to locate plugin: devicemapper, retrying in 2s 
WARN[2018-11-13T10:38:03.522773842-05:00] Unable to locate plugin: devicemapper, retrying in 4s 
^CINFO[2018-11-13T10:38:04.863216849-05:00] Processing signal 'interrupt'                
WARN[2018-11-13T10:38:07.523305063-05:00] Unable to locate plugin: devicemapper, retrying in 8s 
^CINFO[2018-11-13T10:38:08.095129525-05:00] Processing signal 'interrupt'                
^CINFO[2018-11-13T10:38:08.303084600-05:00] Processing signal 'interrupt'                
^CINFO[2018-11-13T10:38:08.558976666-05:00] Processing signal 'interrupt'                
INFO[2018-11-13T10:38:08.559126689-05:00] Forcing docker daemon shutdown without cleanup; 3 interrupts received ```
sboeuf commented 5 years ago

Hi @thaJeztah, what's the status on this issue? Is this resolved now?

amshinde commented 5 years ago

@thaJeztah We rely on the block storage provided by devicemapper storage driver, so would really like to see this resolved.

sboeuf commented 5 years ago

Also, are you planning to support any alternative to devicemapper? Because block device support provides better performances in our case.

thaJeztah commented 5 years ago

I'm not sure what the current status is; I know this was something that was being looked into, but no details yet if they found a solution.

Also, are you planning to support any alternative to devicemapper?

The current roadmap is to complete the migration to using containerd as a runtime, and for image management (see https://github.com/moby/moby/issues/38043). Currently, the functionality of containerd and dockerd overlaps (as in; they both are capable of image-management, and both implement storage drivers). The goal is to hand over the overlapping parts to containerd. Currently containerd has support for overlay, zfs, btrfs, and aufs as storage drivers ("snapshotters") but additional storage drivers could be added upstream in the containerd project. t.b.h., I'm not sure if devicemapper would find its way in there. Maintenance of devicemapper was largely driven by Red Hat (as it was the only option for their kernels), but now that overlay has become mainstream, focus has shifted in that direction.

Because block device support provides better performances in our case.

Do you have more information about your use-case? Is this for writes or reads ? (for writes, the general recommendation is to use a volume, to skip the CoW performance penalty)

sboeuf commented 5 years ago

@thaJeztah

I'm not sure if devicemapper would find its way in there. Maintenance of devicemapper was largely driven by Red Hat (as it was the only option for their kernels), but now that overlay has become mainstream, focus has shifted in that direction.

In our case, we get better performance with block device rootfs. This means that we're not strongly attached to devicemapper itself but to the fact that it exposes block devices. And so it's important for us to keep at least one storage driver that exposes rootfs as block device if we want to maintain this level of performance.

Do you have more information about your use-case?

Because Kata Containers runtime spawn a VM, and because the rootfs needs to be passed inside this VM, we are limited by the device model exposed by QEMU. And in this case, using virtio-block or virtio-scsi is more performant than using 9pfs. But virtio-scsi is block device based... that's the rationale why we need a storage driver exposing rootfs as block device.

thaJeztah commented 5 years ago

Gotcha, thanks for the extra info. /cc @dmcgowan @crosbymichael

maityh commented 5 years ago

Did anyone get rid from this issue? After upgrading the docker system we had faced the same and downgraded the version. Switch to the overlay2 is the only option?

nemesis545 commented 5 years ago

that is the only option, i rollback to previous version just for export a docker image, them upgrade again and import the docker image...

enoch85 commented 5 years ago

This is not a "one-stop-shop" solution, but might help anyone looking for migrating to overlay2.

https://github.com/nextcloud/vm/blob/master/static/docker_overlay2.sh

gerroon commented 5 years ago

This is a wreckless upgrade, how on earth. I had to downgrade too.

burka commented 5 years ago

I would pin the package to 18.06 until this issue has been resolved.

cat > /etc/apt/preferences.d/docker  <<EOF
Package: docker-ce
Pin: version 18.06*
Pin-Priority: 1000
EOF

apt-get install docker-ce 
journalctl -xe | grep docker

I posted this a suggested solution on askubuntu: https://askubuntu.com/a/1111895/24580.

nemesis545 commented 5 years ago

the upgrade's inevitable, the simplest option is a rollback, export, upgrade and import.

akosyrev commented 5 years ago

What is current status of devicemapper? It is removed in build scripts https://github.com/docker/docker-ce-packaging/blob/master/image/Dockerfile.engine#L43 with this PR https://github.com/docker/docker-ce-packaging/pull/223/files#diff-6670e1ec6985ef6cc31ddd74f1e52e70R43
I cannot find any information about deprecation here https://docs.docker.com/storage/storagedriver/device-mapper-driver/ .

Devicemapper is best choice when you need storage driver operating at the block level (when you need to change 1-2 little parts of big file inside container). I know there are ZFS and BTRFS drivers. I am not sure about their reliability, but I know ZFS does not provide DIRECT_IO for now.

@thaJeztah , do you have any information?

thaJeztah commented 5 years ago

@seemethere do you have more info on the status for devicemapper in the .deb packages?

egernst commented 5 years ago

still looking for ETA on fix -- we have plenty of developers who are looking to test Kata with Firecracker, and I am uncomfortable suggesting an old version of Docker who's support window is winding down.

Is there an ETA?

dmandalidis commented 5 years ago

@thaJeztah is this an issue only for .deb packages? In any case, it seems that someone with this problem (.deb or all) needs to upgrade to 18.09.2 (to address the latest CVE) and switch to overlay2 at once.

thaJeztah commented 5 years ago

This is only an issue with .deb packages of 18.09.x. To address the CVE, there's also an updated 18.06 package (18.06.2), which contains a patched version of runc to mitigate the CVE, and should still have support for device mapper in the .deb packages. One restriction for that package, is that the fix for the CVE does not work on Ubuntu 14.04 when running the original 3.13 kernel; before updating the package, upgrade to an Ubuntu provided 4.x kernel for Ubuntu 14.04.

gdahlm commented 5 years ago

Is this feature gone from 18.09? While engine lacks a feature-deprecation-policy on this page, it would be nice if documentation and warnings happened.

https://docs.docker.com/engine/deprecated/#device-mapper-storage-driver

For users who use LVM with low latency NVMe drives with some test cases like DBs Overlay2 is a huge and breaking change. I am extremely disappointed that there is no note that this feature is removed without notification. I get that loopback-lvm performed pretty poorly but it would be nice if depreciation vs removal was accurately documented.

https://docs.docker.com/storage/storagedriver/select-storage-driver/#supported-storage-drivers-per-linux-distribution

²) The devicemapper storage driver is deprecated in Docker Engine 18.09, and will be removed in a future release. It is recommended that users of the devicemapper storage driver migrate to overlay2.

It would be nice to know if I have to find a new backing storage solution for some needs and if this is marked as a removed feature without a deprecation period if that is has been decided.

cpuguy83 commented 5 years ago

I can't get a straight answer on this from Docker-CE package maintainers (all at Docker Inc), it's quite frustrating.

akosyrev commented 5 years ago

@thaJeztah Can someone just remove that tag 'exclude_graphdriver_devicemapper' ?

thaJeztah commented 5 years ago

I think this issue was related to changes in packaging, to facilitate upgrading/updating the engine versions through images (docker engine activate, docker engine update); (I think) removing that tag caused builds to fail, but I don't know if a solution for that was found already.

cpuguy83 commented 5 years ago

So, break existing users for a feature that nobody is using? 👻

thaJeztah commented 5 years ago

FWIW, it's still in the .rpm packages (CentOS, RHEL, Fedora), but not in the .deb (Debian, Ubuntu)

cpuguy83 commented 5 years ago

That just makes no sense to me. Are these new features not supported on CentOS/Fedora/RHEL?

This is a pretty clear cut break with no explanation at all, not even "oh we messed up but aren't going to fix it either", just silence.

andrewhsu commented 5 years ago

Docker CE 18.09.x works with CentOS and Fedora configured with devicemapper. Docker does not qualify Docker CE on RHEL before release, but it is likely to work based on how RHEL is related to CentOS.

We intended to have Docker CE 18.09.x work with Ubuntu and Debian configured with devicemapper as well, but we have encountered complications with the way we build the binaries for the packages that was discovered after release. This is because we did not qualify Docker CE 18.09.x on devicemapper, but on the newer-supported overlay2 which we intend to have as the default, hence the deprecation notice for devicemapper.

Apologies for the problem this has caused existing devicemapper users on Ubuntu and Debian. This is not the way we intend to phase out support for something we know as common use. This is a bug that has definitely slipped through the cracks after the initial 18.09.0 release.

I'll circle back around with the team on what we gotta do to get the devicemapper support back in for 18.09.x on Ubuntu and Debian. In the meantime, if migration is an option, there are instructions for howto convert devicemapper to overlay2, to be performed on your system before upgrade to 18.09.x obviously.

cpuguy83 commented 5 years ago

Thanks for taking a look, @andrewhsu.

gerroon commented 5 years ago

@andrewhsu

I hope you realize that there are a lot of layman users like me out there, I just use Docker when the app developers recommend as a preferred method. I do not do Docker dev or Linux system configurations. I am just a basic user who needs to use Docker because it is recommended. When the system breaks like this in the middle of nothing, it is very hard to make sense of what is going on in these situations. I did not know that my Docker appps all were down, until when I realized that all my web apps were down.

If the migration is the only option then please provide a way to automatically migrate all this stuff.

instantlinux commented 5 years ago

@andrewhsu there are reasons as a non-layman user I explicitly CHOSE to use devicemapper rather than overlay2. None of the documentation I've seen has stated WHY overlay2 is better than devicemapper, or refutes the reasons that I and many thousands of others chose devicemapper.

My reasons? Once configured properly (which isn't that hard, and is easily automated with ansible), the devicemapper thinpool provides what I believe to be the fastest-possible performance. And it enables automatic LVM volume-size extension; I can allocate (say) 10GB on a terabyte physical volume in set-and-forget fashion. The volume might grow to 50GB or 100GB but I can still use the rest for purposes other than docker images.

In a nutshell: I went through the list of available storage drivers circa Feb 2017, chose devicemapper, and have had no reason to reconsider that choice--until the snafu with your company's CI failure in the 18.09 release cycle. And now I have to assume that your managers are busily reprioritizing engineering tasks and aren't likely to ever revisit devicemapper. In any event, before closing this ticket out as won't-fix, please have someone document the advantages of overlay2 better and explain the disadvantages of devicemapper (if any). Because I still don't see why I'm forced to make this change, and I'm concerned about suddenly losing autoextend or I/O performance afterward.

svvac commented 5 years ago

In the meantime, if migration is an option, there are instructions for howto convert devicemapper to overlay2, to be performed on your system before upgrade to 18.09.x obviously.

The referenced procedure doesn't lay out how to migrate an existing devicemapper install. It basically just tells you to scratch your existing storage and create a new one, which I dont think is an acceptable migration procedure in any case.

easp commented 5 years ago

Also an issue on Xenial.

ganeshmaharaj commented 5 years ago

With docker 18.09 using containerd as the runtime handler, is there a way to docker use a storage-driver that exists in containerd? There are multiple efforts to add block device support to containerd here, here and here. It will be nice to allow one of these to be the default storage driver for docker.

thaJeztah commented 5 years ago

With docker 18.09 using containerd as the runtime handler, is there a way to docker use a storage-driver that exists in containerd?

Not possible yet, as those parts are not yet integrated with the docker daemon (storage-drivers are still handled by dockerd). Work is in progress to integrate those parts; see https://github.com/moby/moby/pull/38738 and https://github.com/moby/moby/pull/38925)

seemethere commented 5 years ago

Hey guys!

The latest nightly builds after https://github.com/docker/docker-ce-packaging/pull/314 was merged should contain devicemapper support on Ubuntu and Debian and should be included in the next release for Docker CE!

You can install the latest nightly builds with get.docker.com using:

curl -fsSL get.docker.com | CHANNEL=nightly sh

Ran a quick test with Ubuntu 18.04 to verify:

~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

~$ docker --version
Docker version 0.0.0-20190329010133-83264de7d5, build 83264de7d5

~$ docker info 2>/dev/null | grep Storage
 Storage Driver: devicemapper

Keep in mind that devicemapper will still be deprecated at some point in the future so moving to overlay2 is still preferred.

egernst commented 5 years ago

@seemethere - can the deprecation be held off until https://github.com/docker/for-linux/issues/452#issuecomment-477538868 is addressed?

For some users, overlay2 isn't an option. Once above is addressed, they'll have a viable and supported option.

@thaJeztah WDYT?

thaJeztah commented 5 years ago

Devicemapper will still be there in the upcoming (19.03) release. In addition, work is in progress in containerd to provide block-device snapshotters;

egernst commented 5 years ago

Yep - I just want to make sure containerd snapshotters work with docker (see Ganesh’s prior comment).

megakoresh commented 5 years ago

Even though https://github.com/docker/docker-ce-packaging/pull/314 has been released, this issue is still present for 18.09.5 for yum repositories

Hypocritus commented 5 years ago

I don't understand how a company with as reputable a name as Docker allows an issue this disruptive, first, to happen, or worse, to continue effectively unaddressed and untreated for over six months, despite the caliber of professionals which continue to (for all intents and purposes) politely petition an appropriate and open-minded response or set of solutions.

Attempting to use the "supported" and functional ZFS driver, which unfortunately no longer works due to this change.

Use Overlay2? No! And my reasons are just as valid as the infinitely diverse other reasons had and given by other sysadmins here and elsewhere.

dmcgowan commented 5 years ago

to continue effectively unaddressed and untreated for over six months

Not sure if you read the prior comments, this was addressed and taken seriously. If you are continuing to have a problem after the proposed solution, please highlight what steps you tried and how they have failed.

Use Overlay2? No!

No one is forcing anyone to use overlay2. The devicemapper driver in Docker is being deprecated due to lack of maintenance, it is not being immediately removed. Rather devicemapper support will be done through containerd in the future and in the long term replaces the deprecated graphdriver implementation. I think this wasn't communicated well and the unintentional package breakage gave the impression that they could not longer use devicemapper with Docker anymore. That is not the case, please see the work being done in containerd to support for block device snapshotters.

@thaJeztah @seemethere if this issue has been fixed can you close this issue?

instantlinux commented 5 years ago

@dmcgowan as noted above already, the issue hasn't been fixed in a way that we users have access to. Until a 19.x version comes out, we're having to run back-ported updates of 18.06.x.

seemethere commented 5 years ago

This issue has been resolved in 19.03.X, will close this issue once a GA version of that package is available.

tao12345666333 commented 5 years ago

This issue has been resolved in 19.03.X, will close this issue once a GA version of that package is available.

https://github.com/docker/docker-ce/releases/tag/v19.03.1 19.03 has been released. Should we close this issue?