unable to start containers with existing vsphere volumes after upgrading

[bteichner]

After upgrading from version 0.17 of the plugin to version 0.20, we are unable to start services that had existing vsphere Docker volumes.

The following error is displayed when running docker service ps service-name --no-trunc VolumeDriver.Mount: EOF

I'm also seeing this warning in the Docker daemon logs:

"Error restoring volume\" driver=\"vsphere:latest\" error=\"error while checking if volume \\\"service-name\\\" exists in driver \\\"vsphere:latest\\\": Error looking up volume plugin vsphere:latest: plugin \\\"vsphere:latest\\\" not found\" volume=volume-name"

Docker version: 17.06.2-ee-6

Upgrade Steps Upgrade plugin on esxi hosts

1. esxcli software vib remove -n esx-vmdkops-service
2. esxcli software vib install -v /.vib

Upgrade plugin on Docker hosts

1. docker plugin disable -f vsphere
2. docker plugin upgrade --grant-all-permissions vsphere:latest vmware/vsphere-storage-for-docker:0.20
3. sysetmctl restart docker (required on worker nodes before re-enabling the plugin)
4. docker plugin enable vsphere

[govint]

@bteichner could you also attach the logs from the ESX host (/var/log/vmware/vmdk_ops.log) and from the docker host (/var/log/docker-volume-vsphere.log)

[govint]

@bteichner - could you respond with the logs. Thanks.

[bteichner]

@govint Can I email you the logs, rather than posting them here?

[govint]

@bteichner Thanks for mailing the logs. Looks like the KV file for a container volume is missing. Let me check why this would have happened. But if you have the ESX side logs in its entirety that would help understand why/how the KV file was missing. Can you compress the ESX logs and mail them.

[govint]

From the logs received, one KV file is reported as not present, IOError: [Errno 2] No such file or directory: '/vmfs/volumes/NTNX-NFS-DOCKER-PRD/dockvols/UCPEU_PRD/compasseu-solr-prd-73f3739dc6d0707a.vmfd' 02/05/18 21:01:05 16097762 [PRD-DVT-APP-031-NTNX-NFS-DOCKER-PRD.UCPEU_PRD.compasseu-solr-prd] [ERROR ] Failed to access /vmfs/volumes/NTNX-NFS-DOCKER-PRD/dockvols/UCPEU_PRD/compasseu-solr-prd-73f3739dc6d0707a.vmfd

And confirmed offline that this file is indeed not present in the folder - /vmfs/volumes/NTNX-NFS-DOCKER-PRD/dockvols/UCPEU_PRD.

The logs don't indicate when or why this file was deleted. I'm not sure if we can proceed much further unless we can get a repro of this issue. I'm also checking if #2065 is a similar issue. I'll check the logs once its available there and update here.

[govint]

Change out for review #2067

[shuklanirdesh82]

Please try out the new VIB as mentioned at https://github.com/vmware/vsphere-storage-for-docker/releases/tag/0.21.1