snapshot not labeling volumesnapshot

[JWasdin]

Is this a BUG REPORT or FEATURE REQUEST?: Bug Report

What happened: When trying to take down a deployment we noticed a certain volumesnapshot on a mongodb was not being removed. There was also a host failure on a portworx node during the request to take the volumesnapshots down. On inspection of the logged serialization of the snapshot resource, which was supposed to be deleted, mongodb-2, some of its release labels had been removed. There is only one occurrence of the label in mongodb-2, whereas, in another mongodb node, which did successfully shutdown, there were 3 label references.

We believe there may be a race condition between our labeling of the resource and Stork fetching it from kubernetes. This race condition is fine, except Stork does not do a differential update and so replaces the labels with their old resource version thus preventing us from actions to the resource based on label reference.

What you expected to happen: We would expect the release label would not be overwritten as Stork would do a differential update instead of an overwrite.

How to reproduce it (as minimally and precisely as possible): Create a snapshot, add label immediately after creation, then attempt an update via Stork. See if the initial labels have been maintained on the snapshot or removed by the Stork upgrade.

Anything else we need to know?: This issue request is for Vertiv SRE

Environment:

• Kubernetes version (use kubectl version):

  Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.9+coreos.1", GitCommit:"cd373fe93e046b0a0bc7e4045af1bf4171cea395", GitTreeState:"clean", BuildDate:"2018-03-13T21:28:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Azure

• OS (e.g. from /etc/os-release): coreos

• Kernel (e.g. uname -a): 4.14.32-coreos

• Install tools: Helm

• Others: N/A

[disrani-px]

Filed a bug in kubernetes for this: https://github.com/kubernetes-incubator/external-storage/issues/738 Will submit a fix.

[JWasdin]

Thanks @disrani-px

[cehoffman]

Whats the plan on a new release? If not soon we can look at bringing in a vendor scheme to produce our own builds.

[disrani-px]

@cehoffman New release (v1.1) should be out in a few days