envtest flake: VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists When pvc to be restored has already been created [It] ensure PVC should not fail #1371
The test fails in 2 ways, a GET failure or an UPDATE failure.
The issue seems to be a stale cache, or a stale object that is being updated with annotations.
Core reconciler would try again hence not a code issue per-se, whereas in the tests this is only tried once. Potential fix can leverage error string patterns (like this) to try again (here)to ensure this flake is not seen.
VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists When pvc to be restored has already been created [It] ensure PVC should not fail
.../ramen/controllers/volsync/vshandler_test.go:1329
Timeline >>
2024-05-03T11:57:16.382-0400 INFO Latest Image for ReplicationDestination {"latestImage": "testingsnap001"}
2024-05-03T11:57:16.385-0400 DEBUG VolumeSnapshot validated and protected {"volumesnapshot name": "testingsnap001"}
2024-05-03T11:57:16.385-0400 DEBUG Updated {"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "pvc": {"namespace": "vh-2jx9b", "name": "testpvc1"}}
2024-05-03T11:57:16.388-0400 DEBUG PVC createOrUpdate Complete {"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "op": "created"}
2024-05-03T11:57:16.391-0400 INFO Latest Image for ReplicationDestination {"latestImage": "testingsnap001"}
2024-05-03T11:57:16.391-0400 DEBUG VolumeSnapshot validated and protected {"volumesnapshot name": "testingsnap001"}
2024-05-03T11:57:16.391-0400 DEBUG Updated {"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "pvc": {"apiVersion": "v1", "kind": "PersistentVolumeClaim", "namespace": "vh-2jx9b", "name": "testpvc1"}}
2024-05-03T11:57:16.391-0400 DEBUG PVC createOrUpdate Complete {"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "op": "unchanged"}
2024-05-03T11:57:16.394-0400 DEBUG Failed to Update PVC {"error": "Operation cannot be fulfilled on persistentvolumeclaims \"testpvc1\": the object has been modified; please apply your changes to the latest version and try again"}
[FAILED] in [It] - .../ramen/controllers/volsync/vshandler_test.go:1332 @ 05/03/24 11:57:16.394
<< Timeline
[FAILED] Expected success, but got an error:
<*errors.StatusError | 0xc00139e820>:
Operation cannot be fulfilled on persistentvolumeclaims "testpvc1": the object has been modified; please apply your changes to the latest version and try again
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Operation cannot be fulfilled on persistentvolumeclaims \"testpvc1\": the object has been modified; please apply your changes to the latest version and try again",
Reason: "Conflict",
Details: {
Name: "testpvc1",
Group: "",
Kind: "persistentvolumeclaims",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 409,
},
}
In [It] at: .../ramen/controllers/volsync/vshandler_test.go:1332 @ 05/03/24 11:57:16.394
------------------------------
•••••••••••••
Summarizing 1 Failure:
[FAIL] VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists When pvc to be restored has already been created [It] ensure PVC should not fail
.../ramen/controllers/volsync/vshandler_test.go:1332
VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists [JustBeforeEach] When pvc to be restored has already been created but has incorrect datasource ensure PVC should delete the pvc with incorrect datasource and return err
[JustBeforeEach] .../ramen/controllers/volsync/vshandler_test.go:1207
[It] .../ramen/controllers/volsync/vshandler_test.go:1365
Timeline >>
2024-05-03T11:58:57.583-0400 INFO Latest Image for ReplicationDestination {"latestImage": "testingsnap001"}
2024-05-03T11:58:57.585-0400 DEBUG VolumeSnapshot validated and protected {"volumesnapshot name": "testingsnap001"}
2024-05-03T11:58:57.585-0400 DEBUG Updated {"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "pvc": {"namespace": "vh-247jt", "name": "testpvc1"}}
2024-05-03T11:58:57.587-0400 DEBUG PVC createOrUpdate Complete {"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "op": "created"}
2024-05-03T11:58:57.587-0400 DEBUG Failed to GET PVC {"error": "PersistentVolumeClaim \"testpvc1\" not found"}
[FAILED] in [JustBeforeEach] - .../ramen/controllers/volsync/vshandler_test.go:1209 @ 05/03/24 11:58:57.587
<< Timeline
[FAILED] Unexpected error:
<*fmt.wrapError | 0xc001045960>:
PersistentVolumeClaim "testpvc1" not found
{
msg: "PersistentVolumeClaim \"testpvc1\" not found",
err: <*errors.StatusError | 0xc0002037c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "PersistentVolumeClaim \"testpvc1\" not found",
Reason: "NotFound",
Details: {
Name: "testpvc1",
Group: "",
Kind: "PersistentVolumeClaim",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 404,
},
},
}
occurred
In [JustBeforeEach] at: .../ramen/controllers/volsync/vshandler_test.go:1209 @ 05/03/24 11:58:57.587
------------------------------
•••••••••••••••••••
Summarizing 1 Failure:
[FAIL] VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists [JustBeforeEach] When pvc to be restored has already been created but has incorrect datasource ensure PVC should delete the pvc with incorrect datasource and return err
.../ramen/controllers/volsync/vshandler_test.go:1209
The test fails in 2 ways, a GET failure or an UPDATE failure.
The issue seems to be a stale cache, or a stale object that is being updated with annotations.
Core reconciler would try again hence not a code issue per-se, whereas in the tests this is only tried once. Potential fix can leverage error string patterns (like this) to try again (here)to ensure this flake is not seen.