v1.9.0 - Is this the intended behavior for installation that pvc and pv need to be created manually?

Mia-Dan commented 4 months ago

Validation Checklist

[X] Is this a Kubeflow issue? (not very sure)
[X] Are you posting in the right repository ?
[X] Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ?
[X] Is the issue report properly structured and detailed with version numbers?
[ ] Is this for Kubeflow development ?
[ ] Would you like to work on this issue?
[ ] You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.9

Describe your issue

Hello there,

I'm trying to install Kubeflow v1.9 to local k8s v1.28. While all things work after manually create PV and recreate PVC, I wonder if that is the expected behavior, since in a video tutorial everything is just up and running directly after the scripts is run.

Thanks a lot, Mia

Steps to reproduce the issue

Here's the default storage class configuration I use:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rancher.io/local-path
reclaimPolicy: Delete

# k get storageclasses.storage.k8s.io 
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path   Delete          Immediate           false                  30m

Then run the following line:

while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 20; done

In kubeflow namespace, some pods are complaining:

0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod..

I found creating PV and PVC manually for each of those pods would be a work-around.

Here I do it for katib-mysql.

# k get -n kubeflow pvc
NAME             STATUS    VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
katib-mysql      Bound     katib-mysql   10Gi       RWO            local-path     25h
minio-pvc        Pending                                           local-path     25h
mysql-pv-claim   Pending                                           local-path     25h

And katib is up and running.

Put here any screenshots or videos (optional)

No response

juliusvonkohout commented 3 months ago

@diegolovison is the documentation lead

diegolovison commented 3 months ago

What exactly is local k8s v1.28 ? Do you mean using kind?

Mia-Dan commented 3 months ago

What exactly is local k8s v1.28 ? Do you mean using kind?

No, I was trying to refer to a 3-node k8s cluster set up by kubeasz (which has passed CNCF k8s conformance test), where each node is a VM instance residing in my PC.

Either our local Kind (installed below) or your own Kubernetes cluster with a default StorageClass

Any ideas about it?

juliusvonkohout commented 3 months ago

Well if you are the cluster admin you have to make sure that your cluster also provides storage via dynamic PVs. The PVC is Kubeflows responsibility, the PV is your responsibility.

kubeflow / manifests