SCT tests for Operator are failing during K8s cluster creation while installing dynamic local volume provisioner

What happened?

Example argus run: https://argus.scylladb.com/tests/scylla-cluster-tests/9d9291b1-27b4-4cd4-9e3c-2ca69700b2ae Failing step:

Command: 'kubectl --kubeconfig=/home/ubuntu/sct-results/20241103-010504-970249/functional-master-9d9291b1/.kube/config --cache-dir=/home/ubuntu/sct-results/20241103-010504-970249/.kube/http-cache--functional-master-9d9291b1 --namespace=local-csi-driver rollout status daemonset.apps/local-csi-driver'
Stdout:
Waiting for daemon set spec update to be observed...
Waiting for daemon set "local-csi-driver" rollout to finish: 3 out of 4 new pods have been updated...
Waiting for daemon set "local-csi-driver" rollout to finish: 0 of 4 updated pods are available...

In must-gather logs we can see an error in NodeConfig

  - lastTransitionTime: "2024-11-03T01:27:27Z"
    message: 'MountControllerNodeip-10-0-0-28.eu-north-1.compute.internalDegraded:
      can''t create mounts: can''t ensure units: can''t get unit statuses: can''t
      list units by names "mnt-persistent\\x2dvolumes.mount": Unknown method ''ListUnitsByNames''
      or interface ''org.freedesktop.systemd1.Manager''.'
    observedGeneration: 1
    reason: MountControllerNodeip-10-0-0-28.eu-north-1.compute.internalDegraded_Error
    status: "True"
    type: Nodeip-10-0-0-28.eu-north-1.compute.internalDegraded

What did you expect to happen?

No error

How can we reproduce it (as minimally and precisely as possible)?

You can rerun argus job.

Scylla Operator version

v1.15.0-alpha.0-93-g24fd817-latest

But I have seen that this error start occur since version v1.15.0-alpha.0-56-g8bdba4f-latest

Kubernetes platform name and version

Please attach the must-gather archive.

Must-gather archive can be downloaded from here (Logs tab): https://argus.scylladb.com/tests/scylla-cluster-tests/9d9291b1-27b4-4cd4-9e3c-2ca69700b2ae

Anything else we need to know?

No response

scylladb / scylla-operator

SCT tests for Operator are failing during K8s cluster creation while installing dynamic local volume provisioner #2181