search

NetApp / netappdvp

A Docker volume plugin for NetApp storage
96 stars 33 forks source link

docker cp to attached volume in container causes an unmount to be issued #93

Closed j-griffith closed 7 years ago

j-griffith commented 7 years ago

If you try and use to xfr data from node to a mounted volume in a container the result is an unMount being sent from Docker after the Mount is detected as already existing.

To reproduce: docker volume create -d netapp --name=foobar docker run -it --volume-driver netapp --volume foobar:/data --name crumbs bash

Then on the host: docker cp /var/log/syslog- crumbs:/data/syslog

After running the docker cp command the volume is unmounted by the go-plugin-helper, the logs show this here:

DEBU[2017-08-25T15:21:05Z] Mount(&{foo b995165f99e38f246a0236cda8c2505b5922dc8c0632cc685a28f18ed67f1814}) DEBU[2017-08-25T15:21:05Z] Mounting volume foo on /var/lib/docker-volumes/solidfire/foo DEBU[2017-08-25T15:21:05Z] Begin osutils.GetDFOutput DEBU[2017-08-25T15:21:05Z] /var/lib/docker-volumes/solidfire/foo already mounted, returning existing mount 2017/08/25 15:21:05 Entering go-plugins-helpers unmountPath DEBU[2017-08-25T15:21:05Z] Unmount(&{foo b995165f99e38f246a0236cda8c2505b5922dc8c0632cc685a28f18ed67f1814}) DEBU[2017-08-25T15:21:05Z] SolidfireSANStorageDriver#Detach(foo, /var/lib/docker-volumes/solidfire/foo) DEBU[2017-08-25T15:21:05Z] Begin osutils.Umount: /var/lib/docker-volumes/solidfire/foo DEBU[2017-08-25T15:21:05Z] Response from umount /var/lib/docker-volumes/solidfire/foo:

I think this might be an upstream issue with the Docker volume API but not sure yet.

j-griffith commented 7 years ago

After further investigation this appears to be a problem with Docker and some assumptions that may be made regarding iSCSI connections.

There are different ways to creat an iSCSI target for a device, one is to use unique target IQN's for each "volume" that exists on a device. This is how SolidFire works.

The other is to used a shared TGT IQN and access individual volumes via iscsi portal/lun assignment. A number of devices use this method including the ONTAP devices.

The docker cp operation in this case does some things with a volume that's attached to a container that doesn't work out well for unique targets (SolidFire for example). As part of the copy process docker will first issue an unmount command to the plugin API, which in our case does an iSCSI logout. This is the first problem, this means we just disconnected and unmounted an in use file system. Next, docker will issue a new Mount command; and copy the data from the local system to the mount point on the local system. After that's complete, it will again Unmount, disconnect and then Mount again back on the container.

This won't work for devices that use individual targets and could potentially cause issues for others. To work around this the best option is to mount the data source in to the container as well. From there use the container FS to copy the data from the source to the attached NDVP volume in the container. So something like this: docker run -it -v <path-to-source-data>:/src-data --volume-driver ndvp -v my-ndvp-vol:/dest debian cp /src-data/<my-stuff> /dest/

dutchiechris commented 7 years ago

@j-griffith Eeww! Is there an issue open with the Moby project? I searched but couldn't find one.

j-griffith commented 7 years ago

I have a fix using ref counting here, there's definitely some things we should've been doing better here.

j-griffith commented 7 years ago

Preview of fix here: https://github.com/j-griffith/netappdvp/tree/refcount_attachments_in_solidfire

ntap-rippy commented 7 years ago

The change in https://github.com/NetApp/netappdvp/commit/29aceafd74d82aba6531b8a5c4da0c1a463cd95c should work around the problem by delaying the iSCSI session teardown until we delete the volume. The 'docker cp' commands that failed above work for me after this change.