incorrect status.creationTime value in snapshot creation with useDataPlaneAPI: “true”

What happened:

Hi Team,

I have a customer that had throttling issues when creating snapshots (429 errors).

While investigating, I found a similar case where it was suggested to the customer to use the parameter useDataPlaneAPI: “true” in snapshot class (to use data plane api to take snapshots), that would solve the throttling issue.

The customer followed my suggestion, and it seems to have resolved the throttling issue but there is however a new issue or bug when creating snapshots, having useDataPlaneAPI: “true” in the snapshot class.

Field status.creationTime has an old date. This date is the same for all snapshots of that particular PVC. So, snapshot of other PVC will have different date, but also in the past.

Below is an example of snapshot created today (4.07.2024) at 10:00 UTC, but the status fields show a date from more than two years ago.

kubectl -n confluence-sandbox describe volumesnapshot confluence-sandbox-shared-home-wiki-2hourly-2024-07-04t10-00z 

Name:         confluence-sandbox-shared-home-wiki-2hourly-2024-07-04t10-00z
Namespace:    confluence-sandbox
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2024-07-04T10:00:02Z
[...]
Status:
  Bound Volume Snapshot Content Name:  snapcontent-e9d26b10-2545-4e6e-9f9c-dcd8ebe6a4fa
  Creation Time:                       2022-05-09T15:04:17Z
  Ready To Use:                        true
  Restore Size:                        100Gi
Events:
[...]

If the customer removes the useDataPlaneAPI parameter, then things go back to normal and status.creationTime shows correct date.

Is this behavior normal?

Also, I'm hoping someone can help me answer some questions raised by the customer, as stated below:

“Using Azure Portal, I tried to find this old date in the Storage Account, in the snapshot properties. But I did not find it. So it seems this information is stored only on the Kubernetes side. Can you confirm that?”
“Can you confirm that in the example above, snapshot content is actually from 2024-07-04T10:00:02Z, and not suddenly somehow more than two years old?”
“Can you please check with the product team if they are aware of this bug and whether it has any serious negative consequences? Most importantly, is there any logic, automatic decisions performed by AKS and/or Storage Account based on the information in that field?”

I would appreciate your help on this.

What you expected to happen:

How to reproduce it:

Anything else we need to know?:

use temSnapshotTime = properties.Date() may solve the problem

/kind bug

Environment:

CSI Driver version:
Kubernetes version (use kubectl version):
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

kubernetes-sigs / azurefile-csi-driver

incorrect status.creationTime value in snapshot creation with useDataPlaneAPI: “true” #1957