v1.3.3: SolidFire create/delete/create of volume with same name fails

NetApp / netappdvp

A Docker volume plugin for NetApp storage

96 stars 33 forks source link

v1.3.3: SolidFire create/delete/create of volume with same name fails #71

Closed dutchiechris closed 7 years ago

dutchiechris commented 7 years ago

If you create a volume, delete it, and create it again (same name) it fails to create the volume. I purged the volume from deleted volumes and that didn't help either.

3-docker10 netappdvp # docker volume create -d netapp --name ohno
ohno

3-docker10 netappdvp # docker volume ls
DRIVER              VOLUME NAME
netapp              ohno

3-docker10 netappdvp # docker volume rm ohno
ohno

3-docker10 netappdvp # docker volume ls
DRIVER              VOLUME NAME

3-docker10 netappdvp # docker volume create -d netapp --name ohno
ohno

3-docker10 netappdvp # docker volume ls
DRIVER              VOLUME NAME

YvosOnTheHub commented 7 years ago

2 comments:

when trying to recreate the volume, nothing is written in the /var/log/netappdvp/netapp.log file
the process works with ontap nas

adkerr commented 7 years ago

Internally tracked by DE285

j-griffith commented 7 years ago

Strange, easy enough to reproduce 100% of the time. It seems there's something squirreled away somewhere in the Docker volume side that makes Docker think the volume is still there. It's simply just doing a "Path" command, no Get, no Create etc.

YvosOnTheHub commented 7 years ago

another test I did:

host1: Docker create & delete myVol : OK
host2: Docker create myVol: Failed maybe something to look at: when deleting a SolidFire volume with Docker Volume rm, ElementOS does not immediately delete it, but puts it in a "garbage" that you need to flush eventually... flushing did fix the problem, but not 100% of the time (more like 50%)

j-griffith commented 7 years ago

Yeah, the volume in deleted is "ok" we do handle that; there's something weird going on elsewhere though, because as I said it's like Docker thinks it already knows about the volume. You can verify this also by manually purging the deleted volume off the cluster before trying the create again.

I will trace through the layers of the docker_driver in NDVP and the SolidFire code when I get a second. It's most likely something on the delete response but not sure.

clintonk commented 7 years ago

I’ve seen this with the SolidFire plug-in, but I don’t think I’ve seen it with ONTAP or E-series, so I’d suspect a plug-in issue initially.

j-griffith commented 7 years ago

In fact there are a couple of things, and I stand corrected, you were right when you mentioned the purge issue. Turns out that the reason I was still seeing the issue before was environmental. We really need a better method for running ndvp cleanly.

Anyway, fresh env, and another look it is infact related to the volume not being purged. I added a purge to the DeleteVolume in SolidFire, and will get that pushed up against the internal code repo. In the meantime if you want to mess with it:

https://gist.github.com/j-griffith/12f7c8df864203b9e4d218dff042243d

adkerr commented 7 years ago

Fixed in 1ee9e36