azcopy can't resume cloning job if controller pod is killed during job run

[RomanBednar]

What happened:

While testing azure file cloning in OpenShift we noticed that if a controller pod running azcopy job is killed any clone PVC still being copied gets stuck in Pending phase.

This seems to be the effect of azcopy relying on job plan files (AZCOPY_JOB_PLAN_LOCATION or ~/.azcopy by default) to track jobs which can be lost if stored on ephemeral volume.

Is this a know limitation or is there a recommended solution?

What you expected to happen:

Clone job surviving a lost controller pod.

How to reproduce it:

1) Create Azure File PVC/PV 2) Create a new clone PVC referencing the origin PVC as a source 3) Kill Azure File leader controller pod 4) Clone PVC is stuck in Pending state

Anything else we need to know?:

Checking the helm charts in this repo the destination for those job plan files seems to be ephemeral with emptyDir volume so the issue would occur with this deployment as well: https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/eefed914745603ad28eac52f07c47b972ad3dadd/charts/v1.30.2/azurefile-csi-driver/templates/csi-azurefile-controller.yaml#L229

Environment:

• CSI Driver version:
• Kubernetes version (use kubectl version):
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[andyzhangx]

yes, that's a limitation since the job status can only be stored locally in ~/.azcopy, as long as the controller pod is restarted, copy jobs could not be continued.