Ensure PVC resize does not require backing up the PV metadata again

ShyamsundarR commented 3 years ago

The current assumption is that PVC will bind to the restored PV, even when PVC size is different than PV size (assuming resize happened on cluster1 and workload is moving to cluster2 now).

If PVC fails to bind as the sizes are different, we would need to ensure that the PV is backed up again on resize events (in reality appropriate PVC resize events).

There would also be a race here, where PVC resize completes but PV is not yet backed up and we need to failover, which may need some thought if the problem as stated above exists.

ShyamsundarR commented 3 years ago

We need this tested and confirmed, @raghavendrabhat can you take this up?

raghavendrabhat commented 3 years ago

As of now, VRG gets requeue event notifications whenever the PVC spec changes (which would happen when PVC size is changed), hence I would expect the updated PV metadata to be uploaded by VRG.

However the race between PVC size change and a failover happening before VRG could upload the modified PV metadata is something that can lead to the same situation where an older PV metadata is applied to the new home cluster and PVC size would be different than PV size.

Will try this out.

ShyamsundarR commented 3 years ago

As of now, VRG gets requeue event notifications whenever the PVC spec changes (which would happen when PVC size is changed), hence I would expect the updated PV metadata to be uploaded by VRG.

We add an annotation to not repeat PVC protection steps, so we will not backup the new PV, which hence will have the size as the one it started with.

However the race between PVC size change and a failover happening before VRG could upload the modified PV metadata is something that can lead to the same situation where an older PV metadata is applied to the new home cluster and PVC size would be different than PV size.

Will try this out.

Ack!

raghavendrabhat commented 3 years ago

If PVC is resized (expanded to be more precise) this is what happens

While processing VRG as primary, it checks whether PVC has been protected or not by looking at annotation (which would have been put the first time PVC was successfully processed)
Thus new PV metadata reflecting the resize (i.e. expansion) would not be backed up
At the time of failover, VRG on the new primary would restore the PVs from S3 store which has older size
Now, when the application is deployed, pvc creation will check whether a PV with matching size exists or or not (this is done as part of dynamic provisioning). Since the existing PV size is less than the new PVC size (which was expanded), instead of using the existing PV, a new PV is dynamically provisioned. i.e. in short the restored PV is not bound to the PVC when the application is deployed. Instead a new PV is dynamically provisioned.

However if the PV to be restored is modified to match the new size of the PVC, then it is properly bound to the PVC when the application is deployed.

ShyamsundarR commented 3 years ago

PVCs can be resized at the git or declarative source before being applied to the cluster where the workload is running. The race to backup the PV to the right size would still have corner cases in it that cannot be avoided.

Further, during PV cluster data restore on a managed cluster the PVC size is unknown (as PVC does not exist yet), and hence cannot be validated and ensured to be correct.

This possibly requires some form of replicationHandle exchange than PV cluster data exchange, such that a PVC is created from a VolumeReplication resource, and resize is acted on again on the target cluster when the workload is shifted to cover the corner cases.

ShyamsundarR commented 3 years ago

With the potential scheme of backing up PVCs as well as proposed by @BenamarMk , to address operator created and randomly named StatefulSets (CrunchyDB operator), it maybe possible to address the size discrepancy as well.

The PVC copy with the PV can help with resize, as long as we ensure PVC size and PV size when backing them up are the same (or PV can be larger). So when the application PVC from git is merged to the restored PVC instance, it will be resized via normal processes, but initial claimRef binding will hold as the sizes are compatible.

RamenDR / ramen

Ensure PVC resize does not require backing up the PV metadata again #74