rbd: add support for flattenMode option for replication

Rakshith-R commented 2 weeks ago

This commit adds support for flattenMode option for replication. If the flattenMode is set to "force" in volumereplicationclass parameters, cephcsi will add a task to flatten the image if it has parent. This enable cephcsi to then mirror such images after flattening them. The error message when the image's parent is in trash or unmirrored is improved as well.

Checklist:

[ ] Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
[ ] Reviewed the developer guide on Submitting a Pull Request
[ ] Pending release notes updated with breaking and/or notable changes for the next major release.
[ ] Documentation has been updated, if necessary.
[ ] Unit tests have been added, if necessary.
[ ] Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of the following bot commands in an otherwise empty comment in this PR: * `/retest ci/centos/`: retest the `` after unrelated failure (please report the failure too!)

test results

Restored PVC

alive snapshot:

I0619 09:20:05.482558       1 utils.go:198] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC call: /replication.Controller/EnableVolumeReplication
I0619 09:20:05.485272       1 utils.go:199] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC request: {"parameters":{"schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"}
I0619 09:20:05.506278       1 omap.go:89] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 got omap values: (pool="replicapool", namespace="", name="csi.volume.ae66b7a1-21d7-4205-bc0a-d620e1fbab18"): map[csi.imageid:11c0103d7da8 csi.imagename:csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 csi.volname:pvc-cf11e3a2-50c8-438a-8911-958e046772f8 csi.volume.owner:rook-ceph]
W0619 09:20:05.845510       1 replication.go:153] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot)
I0619 09:20:05.845583       1 replication.go:129] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 flattenMode is not set in parameters, setting to default (never)
E0619 09:20:06.336891       1 replication.go:314] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 failed to enable mirroring on image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18":parent image "replicapool/csi-snap-ad971f5b-61a2-4bf4-8c0b-4ec61cec216e" is not enabled for mirroring
E0619 09:20:06.340126       1 utils.go:203] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC error: rpc error: code = Internal desc = failed to enable mirroring on image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18": parent image "replicapool/csi-snap-ad971f5b-61a2-4bf4-8c0b-4ec61cec216e" is not enabled for mirroring

deleted snapshot

I0619 09:21:45.816334       1 utils.go:198] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC call: /replication.Controller/EnableVolumeReplication
I0619 09:21:45.819942       1 utils.go:199] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC request: {"parameters":{"schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"}
I0619 09:21:45.828980       1 omap.go:89] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 got omap values: (pool="replicapool", namespace="", name="csi.volume.ae66b7a1-21d7-4205-bc0a-d620e1fbab18"): map[csi.imageid:11c0103d7da8 csi.imagename:csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 csi.volname:pvc-cf11e3a2-50c8-438a-8911-958e046772f8 csi.volume.owner:rook-ceph]
W0619 09:21:45.958255       1 replication.go:153] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot)
I0619 09:21:45.958630       1 replication.go:129] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 flattenMode is not set in parameters, setting to default (never)
E0619 09:21:46.093419       1 replication.go:314] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 system is not in a state required for the operation's execution: failed to enable mirroring on image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18": parent is in trash
E0619 09:21:46.093596       1 utils.go:203] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC error: rpc error: code = FailedPrecondition desc = system is not in a state required for the operation's execution: failed to enable mirroring on image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18": parent is in trash

with flattenMode force

I0619 09:24:53.228433       1 utils.go:198] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC call: /replication.Controller/EnableVolumeReplication
I0619 09:24:53.228743       1 utils.go:199] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC request: {"parameters":{"flattenMode":"force","schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"}
I0619 09:24:53.237278       1 omap.go:89] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 got omap values: (pool="replicapool", namespace="", name="csi.volume.ae66b7a1-21d7-4205-bc0a-d620e1fbab18"): map[csi.imageid:11c0103d7da8 csi.imagename:csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 csi.volname:pvc-cf11e3a2-50c8-438a-8911-958e046772f8 csi.volume.owner:rook-ceph]
W0619 09:24:53.387365       1 replication.go:153] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot)
I0619 09:24:53.538514       1 rbd_util.go:829] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 rbd: adding task to flatten image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"
E0619 09:24:53.736260       1 replication.go:314] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 flatten in progress: flatten is in progress for image csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18
E0619 09:24:53.736433       1 utils.go:203] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC error: rpc error: code = Aborted desc = flatten in progress: flatten is in progress for image csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18
...
I0619 09:24:55.058061       1 utils.go:198] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC call: /replication.Controller/EnableVolumeReplication
I0619 09:24:55.058184       1 utils.go:199] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC request: {"parameters":{"flattenMode":"force","schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"}
I0619 09:24:55.090814       1 omap.go:89] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 got omap values: (pool="replicapool", namespace="", name="csi.volume.ae66b7a1-21d7-4205-bc0a-d620e1fbab18"): map[csi.imageid:11c0103d7da8 csi.imagename:csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 csi.volname:pvc-cf11e3a2-50c8-438a-8911-958e046772f8 csi.volume.owner:rook-ceph]
W0619 09:24:55.170256       1 replication.go:153] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot)
I0619 09:24:56.210444       1 utils.go:205] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC response: {}

PVC-PVC clone

with flattenMode default never:

I0619 09:29:13.460421       1 utils.go:198] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 GRPC call: /replication.Controller/EnableVolumeReplication
I0619 09:29:13.468826       1 utils.go:199] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 GRPC request: {"parameters":{"schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7"}
I0619 09:29:13.491370       1 omap.go:89] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 got omap values: (pool="replicapool", namespace="", name="csi.volume.f3f494a9-c9f9-4734-b9e4-b04cd58cfac7"): map[csi.imageid:187bfd24f7c6 csi.imagename:csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 csi.volname:pvc-12fe2ae9-cb90-4bee-9247-6146f11bf8e2 csi.volume.owner:rook-ceph]
W0619 09:29:13.620327       1 replication.go:153] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot)
I0619 09:29:13.620533       1 replication.go:129] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 flattenMode is not set in parameters, setting to default (never)
E0619 09:29:14.225457       1 replication.go:314] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 failed to enable mirroring on image "replicapool/csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7":parent image "replicapool/csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7-temp" is not enabled for mirroring
E0619 09:29:14.226126       1 utils.go:203] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 GRPC error: rpc error: code = Internal desc = failed to enable mirroring on image "replicapool/csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7": parent image "replicapool/csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7-temp" is not enabled for mirroring

with flattenMode force

I0619 10:17:48.127012       1 utils.go:198] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC call: /replication.Controller/EnableVolumeReplication
I0619 10:17:48.127174       1 utils.go:199] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC request: {"parameters":{"flattenMode":"force","schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93"}
I0619 10:17:48.139129       1 omap.go:89] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 got omap values: (pool="replicapool", namespace="", name="csi.volume.d73378c1-62b3-454e-8646-dae3207e7f93"): map[csi.imageid:1d39997d6758 csi.imagename:csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93 csi.volname:pvc-95b1e8ca-590d-420e-9c93-d6686e857d2a csi.volume.owner:rook-ceph]
W0619 10:17:48.284313       1 replication.go:153] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot)
I0619 10:17:48.468021       1 rbd_util.go:634] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 rbd: delete csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93-temp using mon 192.168.122.191:6789, pool replicapool
I0619 10:17:48.494034       1 rbd_util.go:676] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 rbd: adding task to remove image "replicapool/csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93-temp" with id "1d39206cc57d" from trash
I0619 10:17:48.503025       1 rbd_util.go:700] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 rbd: successfully added task to move image "replicapool/csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93-temp" with id "1d39206cc57d" to trash
I0619 10:17:48.503080       1 rbd_util.go:850] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 rbd: adding task to flatten image "replicapool/csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93"
E0619 10:17:48.599921       1 replication.go:314] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 flatten in progress: flatten is in progress for image csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93
E0619 10:17:48.600632       1 utils.go:203] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC error: rpc error: code = Aborted desc = flatten in progress: flatten is in progress for image csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93
...
0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC call: /replication.Controller/EnableVolumeReplication
I0619 10:17:57.469112       1 utils.go:199] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC request: {"parameters":{"flattenMode":"force","schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93"}
I0619 10:17:57.481057       1 omap.go:89] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 got omap values: (pool="replicapool", namespace="", name="csi.volume.d73378c1-62b3-454e-8646-dae3207e7f93"): map[csi.imageid:1d39997d6758 csi.imagename:csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93 csi.volname:pvc-95b1e8ca-590d-420e-9c93-d6686e857d2a csi.volume.owner:rook-ceph]
W0619 10:17:57.530084       1 replication.go:153] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot)
I0619 10:17:57.603214       1 utils.go:205] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC response: {}

mergify[bot] commented 2 weeks ago

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

Rakshith-R commented 1 week ago

I've added testing results in pr description. The pr is ready for review.

iPraveenParihar commented 1 week ago

I've added testing results in pr description. The pr is ready for review.

@Rakshith-R, I don't see the testing results in the PR description. can you re-check if you have missed it?

Rakshith-R commented 1 week ago

I've added testing results in pr description. The pr is ready for review.

@Rakshith-R, I don't see the testing results in the PR description. can you re-check if you have missed it?

forgot to press update button. Its updated now. ptal

Rakshith-R commented 1 week ago

@mergifyio queue

mergify[bot] commented 1 week ago

queue