search

canonical / kubeflow-examples

Charmed Kubeflow examples
Apache License 2.0
32 stars 9 forks source link

`deploy-kf-on-ck8s-aws` failing steps if Charmed K8s is not ready #35

Open phoevos opened 1 year ago

phoevos commented 1 year ago

The notebook guide on deploying Charmed Kubeflow on Charmed Kubernetes (deploy-kf-on-ck8s-aws.ipynb) includes the following steps to deploy and configure the Kubernetes cluster:

juju deploy ./bundle/releases/1.24/bundle.yaml --overlay ./bundle/overlays/aws-storage-overlay.yaml --trust
juju scp kubernetes-control-plane/0:config ~/.kube/config
juju add-k8s charmed-k8s-aws --controller $(juju switch | cut -d: -f1) --storage=cdk-ebs
juju-wait -m default -t 3600

Issue

The commands run in this order won't work, since scp assumes that the charm has been deployed successfully. Executing it earlier will likely result in one of the two following issues:

  1. Config file not yet created (scp run too soon): 
    ERROR exit status 1 (scp: config: No such file or directory)

  2. Config file changed during the installation. This issue won't be instantly evident, but can be surfaced by:

    • Running any kubectl command:
    $ kubectl get ns
E0608 17:52:43.337190    5320 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
    • Proceeding to run the juju add-k8s command:
    $ juju add-k8s charmed-k8s-aws --controller $(juju switch | cut -d: -f1) --storage=cdk-ebs
ERROR making juju admin credentials in cluster: ensuring cluster role "juju-credential-0cf57eda" in namespace "kube-system": Unauthorized

Solution

The solution is actually pretty simple: wait for the deployment to complete before attempting to configure K8s:

juju deploy ./bundle/releases/1.24/bundle.yaml --overlay ./bundle/overlays/aws-storage-overlay.yaml --trust
juju-wait -m default -t 3600
juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config
juju add-k8s charmed-k8s-aws --controller $(juju switch | cut -d: -f1) --storage=cdk-ebs