[lab2] PVC allocation fails with hostpath provisioner

[codificat]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Followed the steps in lab2 on a minikube environment, and the fedora PVC was stuck in Pending state - and so was the importer pod.

What you expected to happen:

Lab steps should work flawlessly.

Anything else we need to know?:

If the issue is with a lab, please provide information about your environment, platform, versions, ...

Running this on a Fedora 33 host:

$ minikube version
minikube version: v1.17.1
commit: 043bdca07e54ab6e4fc0457e3064048f34133d7e
$ kubectl logs -n kube-system hostpath-provisioner-5f8dd4968f-wccrs | tail -4
I0222 18:42:00.072141       1 leaderelection.go:154] attempting to acquire leader lease...
I0222 18:42:00.098092       1 controller.go:726] claim default/fedora was deleted
E0222 18:42:00.113628       1 leaderelection.go:271] Failed to update lock: Operation cannot be fulfilled on persistentvolumeclaims "fedora": StorageError: invalid object, Code: 4, Key: /registry/persistentvolumeclaims/default/fedora, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d567e6df-6051-458a-8b65-a7bbaed82310, UID in object meta: 
I0222 18:42:02.817551       1 leaderelection.go:161] stopped trying to acquire lease to provision for pvc default/fedora, task succeeded

Note that in minikube there is a default standard storage provisioner out of the box, using hostpath.

[mazzystr]

@cwilkers tagged

[cwilkers]

I tried with the latest minikube, which deploys v1.20.2.

This brings up a new error from the hostpath-provisioner controller:

I0325 13:27:04.034431       1 controller.go:893] scheduleOperation[provision-default/fedora[a4684013-abf2-4d3b-b6ab-089d73de1c0a]]
E0325 13:27:04.042861       1 controller.go:592] Unexpected error getting claim reference to claim "default/fedora": selfLink was empty, can't make reference

Re-enabling the self-link feature gate when deploying Minikube seems to be the work-around:

--feature-gates=RemoveSelfLink=false

See https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/issues/25 for a similar issue

[cwilkers]

I was not able to duplicate the problem using minikube v1.17.1 so I will try to fix the bug I found on the current version. (v1.20.2)

[mazzystr]

@cwilkers Are you seeing the bug in latest minikube / kubevirt?

[cwilkers]

On latest minikube / kubernetes / kubevirt (v1.18.1 / v1.20.2 / v1.32.0) I am seeing the above issue, but when we eliminate hostpath provisioner and use the minikube default storage class, "standard" the pvc creates with no issue.

After modifying the PVC to use the standard storage class, there is still an issue with the importer pod where it mishandles the URL's mirror redirect (302). I am still investigating that.

[cwilkers]

Mirror redirect issue seems related to Fedora's own mirrors. As explained in https://github.com/kubevirt/containerized-data-importer/issues/1737 , CDI is not checking links before running a parallel download.

[cwilkers]

I am working on a PR to the lab to switch to using Minkube's own hostpath storageclass, will link here when ready.

[gpierard]

I had the same problem in sept 2023. In Lab 2 using minikube, the below creates the datavolume but no PVC, pods or importers are created.

cat <<EOF > dv_fedora.yml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: "fedora"
spec:
  storage:
    resources:
      requests:
        storage: 5Gi
  source:
    http:
      url: "https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.raw.xz"
EOF

kubectl create -f dv_fedora.yml
kubectl get pvc fedora -o yaml # no resources found
kubectl get pod # no resources found in default namespace
kubectl logs -f importer-fedora-pnbqh   # not found

One solution is to use the pvc definition from the video (presumably related to an older version of lab2) instead of the above DataVolume definition

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "fedora"
  labels:
    app: containerized-data-importer
  annotations:
    cdi.kubevirt.io/storage.import.endpoint: "https://download.fedoraproject.org/pub/fedora/linux/releases/30/Cloud/x86_64/images/Fedora-Cloud-Base-30-1.2.x86_64.raw.xz"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: standard

kubectl create -f pvc_fedora_fromvideo.yml # successfully creates the objects

However, the download still fails for some reason.

kubectl logs -f importer-fedora   
I0913 16:14:13.450381       1 importer.go:103] Starting importer
I0913 16:14:13.450515       1 importer.go:172] begin import process
E0913 16:14:14.394715       1 http-datasource.go:399] http: expected status code 200, got 404

[aburdenthehand]

@gpierard It looks like that release (30) has been archived. I should be able to pull it instead from the archive: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/Cloud/x86_64/images/Fedora-Cloud-Base-30-1.2.x86_64.raw.xz

Of course, this doesn't help with why the original 37 disk didn't create a pvc/importer

[cwilkers]

In debugging the DataVolume version, it is useful to use describe to get a readout of its status. If it is somehow failing to create the PVC, that would become clearer there.