DeleteVolume success/failure if volume is replicated and marked "Secondary"

[ShyamsundarR]

NOTE: This is specific to Ceph-CSI, but looking at what the generic rule would be from a VolumeReplication perspective

Currently if a volume is demoted or marked Secondary from the VolumeReplication CR perspective, and a subsequent delete volume request is sent, it fails to delete the volume and returns errors to the k8s API server. Thus the PV is never garbage collected on a Secondary instance.

The Volume replication and remote volume is managed internal to the storage system and, as it currently stands, the garbage collection of a remote Secondary is also a storage system responsibility. Thus, the expectation is that, when a PV is deleted, but is internally Secondary on the system, the delete volume succeeds and the backing volume is finally garbage collected once the Primary instance is deleted.

If the above holds true, request that ceph-csi start checking image info/status and determine if it is mirrored and secondary to return required success for such images. I would assume ceph-csi still garbage collects the OMaps as that is not cleared when the remote image is deleted, and a DeleteVolume would be the last opportunity to do so on the local storage instance.

[Madhu-1]

@ShyamsundarR I was thinking about the below items to keeps the things generic

• When VR is created add finalizers for the PVC to avoid deleting the PVC if the replication is enabled
• When VR is deleted send DisableReplication Request and remove finalizers (revert back https://github.com/csi-addons/volume-replication-operator/pull/94)
  • If the storage does not allow disabling let the VR and PVC stay in Deleting state and Reconciliation will take care of removing it
• If someone wants to remove the resources after doing failback from a cluster. They can simply send a request to delete the resources and let the Reconciler take care of deleting them.

With the above approach, I think most of the storage providers will take care of handling things internally

Now from the cephcsi side if the image is in the secondary state the image cannot be disabled and also it cannot be deleted. pre-req for Delete and Disable is that image needs to be in a primary state.

[ShyamsundarR]

@ShyamsundarR I was thinking about the below items to keeps the things generic

* When VR is created add finalizers for the PVC to avoid deleting the PVC if the replication is enabled

* When VR is deleted send DisableReplication Request and remove finalizers  (revert back [replication: disable replication if its primary state #94](https://github.com/csi-addons/volume-replication-operator/pull/94))

Why will a Secondary image be disabled for replication? The primary controls the operation, right? So again, disable as secondary should be a no-op in this case.

  * If the storage does not allow disabling let the VR and PVC stay in Deleting state and Reconciliation will take care of removing it

* If someone wants to remove the resources after doing failback from a cluster. They can simply send a request to delete the resources and let the Reconciler take care of deleting them.

Which reconciler would act on the delete?

With the above approach, I think most of the storage providers will take care of handling things internally

Now from the cephcsi side if the image is in the secondary state the image cannot be disabled and also it cannot be deleted. pre-req for Delete and Disable is that image needs to be in a primary state.

This will not work for other orchestration services. For ceph-csi as a result VR cannot be deleted when image is secondary (as disable will fail), which will result in PVC not being deleted as VR finalizer would hold it in place. What is the recourse for the user to ensure these can be deleted?

The ground rules are that storage manages the life cycle of the volume on the Secondary. So deletion when secondary should succeed. We cannot let PVs and PVCs linger on the Secondary, as these need to be cleaned up for a future restore of the workload.

[Madhu-1]

@ShyamsundarR I was thinking about the below items to keeps the things generic

* When VR is created add finalizers for the PVC to avoid deleting the PVC if the replication is enabled

* When VR is deleted send DisableReplication Request and remove finalizers  (revert back [replication: disable replication if its primary state #94](https://github.com/csi-addons/volume-replication-operator/pull/94))

Why will a Secondary image be disabled for replication? The primary controls the operation, right? So again, disable as secondary should be a no-op in this case.

Yes or else we will end up in other problems if both the

  * If the storage does not allow disabling let the VR and PVC stay in Deleting state and Reconciliation will take care of removing it

* If someone wants to remove the resources after doing failback from a cluster. They can simply send a request to delete the resources and let the Reconciler take care of deleting them.

Which reconciler would act on the delete?

PVC Reconciler and VR Reconciler

With the above approach, I think most of the storage providers will take care of handling things internally Now from the cephcsi side if the image is in the secondary state the image cannot be disabled and also it cannot be deleted. pre-req for Delete and Disable is that image needs to be in a primary state.

This will not work for other orchestration services. For ceph-csi as a result VR cannot be deleted when image is secondary (as disable will fail), which will result in PVC not being deleted as VR finalizer would hold it in place. What is the recourse for the user to ensure these can be deleted?

If the images are secondary on both sites the VR, PVC deletion fails and as the VR deletion fails, Now he will know the reason and he makes the VR state as primary and Delete the VR and PVC

The ground rules are that storage manages the life cycle of the volume on the Secondary. So deletion when secondary should succeed. We cannot let PVs and PVCs linger on the Secondary, as these need to be cleaned up for a future restore of the workload.

This is something the orchestration services decide and to clean up the resources. As I mentioned above only the volume replication is stopped the image will get auto-deleted and the corresponding VR and PVC and PV will also get deleted

[Madhu-1]

update from the meeting

Volume replication

• [x] Add finalizer to the PVC when VR is created and delete the finalizers when VR is deleted
• [x] Revert back https://github.com/csi-addons/volume-replication-operator/pull/94

CSI

• [ ] Garbage collect the OMAP objects when PVC is deleted and the image is healthy secondary(skip image deletion)
• [ ] Return success for DisableReplcation if the local image is healthy secondary (let the VR deletion on healthy primary cleanup the image on the remote site)

@ShyamsundarR anything missed?

[ShyamsundarR]

update from the meeting

Volume replication

* Add finalizer to the PVC when VR is created and delete the finalizers when VR is deleted

* Revert back [replication: disable replication if its primary state #94](https://github.com/csi-addons/volume-replication-operator/pull/94)

CSI

* Garbage collect the OMAP objects when PVC is deleted and the image is healthy secondary(skip image deletion)

* Return success for DisableReplcation if the local image is healthy secondary (let the VR deletion on healthy primary cleanup the image on the remote site)

@ShyamsundarR anything missed?

Ack!

Behavior would be as follows:

• VR deleted as Secondary would call disable to storage provider (ensuring storage provider is in Secondary state, i.e it should ensure on delete that storage provider state is the same as VR spec.state, so that storage provider can take the right action)
  • Storage provider decides to do nothing, as image is secondary and hence actions of disable are a No-OP
• VR deletion triggers PVC finalizer deletion, and possibly PVC deletion, leading to PV deletion
  • Storage provider decides to garbage collect any artifacts, and possibly excluds garbage collecting the image as it is a "healthy" secondary and will be garbage collected by the primary instance

[Madhu-1]

CSI PR is up for review. once its merged will close this one.

[Madhu-1]

all the required PR's are merged!! closing it.