search

kubeflow / kubeflow

Machine Learning Toolkit for Kubernetes
https://www.kubeflow.org/
Apache License 2.0
14.43k stars 2.42k forks source link

Error: failed to apply: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 400 with message: couldn't create namespace kubeflow Error: the server could not find the requested resource (post namespaces) #4787

Closed Dhanunjaya-Elluri closed 3 years ago

Dhanunjaya-Elluri commented 4 years ago

When I tried to install Kubeflow on existing cluster k3s usign the documentation https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/

It is for k8s. But trying to work on k3s. I'm getting the below error. Please guide what to do.

$ kfctl apply -V -f kfctl_k8s_istio.v1.0.0.yaml INFO[0000] No name specified in KfDef.Metadata.Name; defaulting to my_kubeflow based on location of config file: kfctl_k8s_istio.v1.0.0.yaml. filename="coordinator/coordinator.go:202" INFO[0000]

Notice anonymous usage reporting enabled using spartakus To disable it If you have already deployed it run the following commands: cd $(pwd) kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/

filename="coordinator/coordinator.go:120" INFO[0000] Deleting cachedir .cache/manifests because Status.ReposCache is out of date filename="kfconfig/types.go:472" INFO[0000] Fetching file:///home/controller/manifests-1.0.0.tar.gz to .cache/manifests filename="kfconfig/types.go:493" INFO[0000] probing file path: ///home/controller/manifests-1.0.0.tar.gz filename="kfconfig/types.go:543" INFO[0000] updating localPath to .cache/manifests/manifests-1.0.0 filename="kfconfig/types.go:552" INFO[0000] Fetch succeeded; LocalPath .cache/manifests/manifests-1.0.0 filename="kfconfig/types.go:561" INFO[0000] Deleting cachedir .cache/kubeflow because Status.ReposCache is out of date filename="kfconfig/types.go:472" INFO[0000] Fetching file:///home/controller/kfctl_v1.0-rc.3-1-g24b60e8_linux.tar.gz to .cache/kubeflow filename="kfconfig/types.go:493" INFO[0001] probing file path: ///home/controller/kfctl_v1.0-rc.3-1-g24b60e8_linux.tar.gz filename="kfconfig/types.go:543" INFO[0001] updating localPath to .cache/kubeflow/kfctl filename="kfconfig/types.go:552" INFO[0001] Fetch succeeded; LocalPath .cache/kubeflow/kfctl filename="kfconfig/types.go:561" INFO[0001] Processing application: istio-crds filename="kustomize/kustomize.go:408" INFO[0001] Processing application: istio-install filename="kustomize/kustomize.go:408" INFO[0001] Processing application: cluster-local-gateway filename="kustomize/kustomize.go:408" INFO[0001] Processing application: istio filename="kustomize/kustomize.go:408" INFO[0001] Processing application: add-anonymous-user-filter filename="kustomize/kustomize.go:408" INFO[0001] Processing application: application-crds filename="kustomize/kustomize.go:408" INFO[0001] Processing application: application filename="kustomize/kustomize.go:408" INFO[0001] Processing application: cert-manager-crds filename="kustomize/kustomize.go:408" INFO[0001] Processing application: cert-manager-kube-system-resources filename="kustomize/kustomize.go:408" INFO[0001] Processing application: cert-manager filename="kustomize/kustomize.go:408" INFO[0001] Processing application: metacontroller filename="kustomize/kustomize.go:408" INFO[0001] Processing application: argo filename="kustomize/kustomize.go:408" INFO[0001] Processing application: kubeflow-roles filename="kustomize/kustomize.go:408" INFO[0001] Processing application: centraldashboard filename="kustomize/kustomize.go:408" INFO[0001] Processing application: bootstrap filename="kustomize/kustomize.go:408" INFO[0001] Processing application: webhook filename="kustomize/kustomize.go:408" INFO[0001] Processing application: jupyter-web-app filename="kustomize/kustomize.go:408" INFO[0001] Processing application: spark-operator filename="kustomize/kustomize.go:408" INFO[0001] Processing application: metadata filename="kustomize/kustomize.go:408" INFO[0001] Processing application: notebook-controller filename="kustomize/kustomize.go:408" INFO[0001] Processing application: pytorch-job-crds filename="kustomize/kustomize.go:408" INFO[0001] Processing application: pytorch-operator filename="kustomize/kustomize.go:408" INFO[0001] Processing application: knative-crds filename="kustomize/kustomize.go:408" INFO[0001] Processing application: knative-install filename="kustomize/kustomize.go:408" INFO[0001] Processing application: kfserving-crds filename="kustomize/kustomize.go:408" INFO[0001] Processing application: kfserving-install filename="kustomize/kustomize.go:408" INFO[0001] Processing application: spartakus filename="kustomize/kustomize.go:408" INFO[0001] Processing application: tensorboard filename="kustomize/kustomize.go:408" INFO[0002] Processing application: tf-job-crds filename="kustomize/kustomize.go:408" INFO[0002] Processing application: tf-job-operator filename="kustomize/kustomize.go:408" INFO[0002] Processing application: katib-crds filename="kustomize/kustomize.go:408" INFO[0002] Processing application: katib-controller filename="kustomize/kustomize.go:408" INFO[0002] Processing application: api-service filename="kustomize/kustomize.go:408" INFO[0002] Processing application: minio filename="kustomize/kustomize.go:408" INFO[0002] Processing application: mysql filename="kustomize/kustomize.go:408" INFO[0002] Processing application: persistent-agent filename="kustomize/kustomize.go:408" INFO[0002] Processing application: pipelines-runner filename="kustomize/kustomize.go:408" INFO[0002] Processing application: pipelines-ui filename="kustomize/kustomize.go:408" INFO[0002] Processing application: pipelines-viewer filename="kustomize/kustomize.go:408" INFO[0002] Processing application: scheduledworkflow filename="kustomize/kustomize.go:408" INFO[0002] Processing application: pipeline-visualization-service filename="kustomize/kustomize.go:408" INFO[0002] Processing application: profiles filename="kustomize/kustomize.go:408" INFO[0002] Processing application: seldon-core-operator filename="kustomize/kustomize.go:408" INFO[0002] .cache/manifests exists; not resyncing filename="kfconfig/types.go:468" INFO[0002] .cache/kubeflow exists; not resyncing filename="kfconfig/types.go:468" INFO[0002] namespace: kubeflow filename="utils/k8utils.go:427" INFO[0002] Creating namespace: kubeflow filename="utils/k8utils.go:432" Error: failed to apply: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 400 with message: couldn't create namespace kubeflow Error: the server could not find the requested resource (post namespaces) Usage: kfctl apply -f ${CONFIG} [flags]

Flags: -f, --file string Static config file to use. Can be either a local path: export CONFIG=./kfctl_gcp_iap.yaml or a URL: export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_existing_arrikto.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml kfctl apply -V --file=${CONFIG} -h, --help help for apply -V, --verbose verbose output default is false

failed to apply: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 400 with message: couldn't create namespace kubeflow Error: the server could not find the requested resource (post namespaces)

Specs:

issue-label-bot[bot] commented 4 years ago

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

jlewi commented 4 years ago

Kubeflow is not intended to run on small edge devices. So if K3s is targeting devices like Rasberry pi this probably won't work.

The error is complaining that it can't find the resource namespaces in your cluster. This seems like an issue with your Kubernetes cluster not Kubeflow.

Can you run 

kubectl get namespaces
kubectl create namespace somenamespace

To see whether namespaces are accessible

jtfogarty commented 4 years ago

/area kustomize

kdunee commented 4 years ago

@DhanunjayElluri remember to set export KUBECONFIG=/etc/rancher/k3s/k3s.yaml when deploying on k3s

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
area/kfctl 1.00

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

tomalbrecht commented 4 years ago

Same problem here, since today. Using microk8s v1.18 and https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml.

Specs:

Kubeflow version: v1.0.2 kfctl version: kfctl v1.0.2-0-ga476281 and kfctl v1.0.1-0-gf3edb9b Kubernetes platform: microk8s v1.18 Kubernetes version: v1.18.4 OS: Ubuntu 18.04 LTS

NFO[0003] updating localPath to /opt/kubeflow/setup/.cache/manifests/manifests-1.0.2  filename="kfconfig/types.go:540"
INFO[0003] Fetch succeeded; LocalPath /opt/kubeflow/setup/.cache/manifests/manifests-1.0.2  filename="kfconfig/types.go:561"
INFO[0003] folder /opt/kubeflow/setup/kustomize exists, skip kustomize.Generate  filename="kustomize/kustomize.go:372"
INFO[0003] /opt/kubeflow/setup/.cache/manifests exists; not resyncing   filename="kfconfig/types.go:468"
INFO[0003] namespace: kubeflow                           filename="utils/k8utils.go:427"
INFO[0003] Creating namespace: kubeflow                  filename="utils/k8utils.go:432"
Error: failed to apply:  (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize:  (kubeflow.error): Code 400 with message: couldn't create namespace kubeflow Error: the server could not find the requested resource (post namespaces)
Usage:
  kfctl apply -f ${CONFIG} [flags]

Flags:
  -f, --file string   Static config file to use. Can be either a local path:
                                export CONFIG=./kfctl_gcp_iap.yaml
                        or a URL:
                                export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml
                                export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.0.yaml
                                export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws.v1.0.0.yaml
                                export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.0.yaml
                        kfctl apply -V --file=${CONFIG}
  -h, --help          help for apply
  -V, --verbose       verbose output default is false

failed to apply:  (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize:  (kubeflow.error): Code 400 with message: couldn't create namespace kubeflow Error: the server could not find the requested resource (post namespaces)
tomalbrecht commented 4 years ago

Update: I had to clean up my environment. Especially I deleted /root/.kube and ~/.kube. Installing microk8s via sudo snap install microk8sseems to mess up the configurations. It will create ~/.kube and set root as owner. Thus kubectl won't fully function as well. After I cleaned the configuration I opened a root shell and reinstalled micro8s and finally deployed kubeflow.

Basically I did:

sudo -i
# cleanup
snap remove microk8s
rm -rf /root/snap/microk8s
rm -rf $HOME/.kube
rm -rf /opt/kubeflow

CHANNEL=${CHANNEL:-1.18/stable}
snap install microk8s --classic --channel=`${CHANNEL}
# ...
$KF_DIR_BIN/kfctl apply -V -f ${CONFIG_FILE}
geewynn commented 4 years ago

Hi @tomalbrecht can I ask if you have a detailed doc on how to deploy kuneflow using https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml

I did all you specified here and encountered an error as I stated in this issue https://github.com/ubuntu/microk8s/issues/1547

attackordie commented 4 years ago

@DhanunjayElluri I had success after following @kdunee reminder to set export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

For others @geewynn interested in adding kubeflow to a k3s cluster I followed the detailed instructions here

Specs:

Kubeflow version: v1.0.2 kfctl version: v1.1.0 Kubernetes platform: k3s OS: Ubuntu 20.04.1 LTS

geewynn commented 4 years ago

@DhanunjayElluri I had success after following @kdunee reminder to set export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

For others @geewynn interested in adding kubeflow to a k3s cluster I followed the detailed instructions here

Specs:

Kubeflow version: v1.0.2 kfctl version: v1.1.0 Kubernetes platform: k3s OS: Ubuntu 20.04.1 LTS

Thanks @attackordie

davidspek commented 3 years ago

Closing this issue as the problem seems to have been solved. /close

google-oss-robot commented 3 years ago

@DavidSpek: Closing this issue.

In response to [this](https://github.com/kubeflow/kubeflow/issues/4787#issuecomment-803381986): >Closing this issue as the problem seems to have been solved. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.