ERROR resolving microk8s credentials: ensuring cluster role “juju-credential-microk8s” in namespace “kube-system”

[dominikkubik]

Please, any idea what Am i doing wrong to have it running as described in official quickstart guide on completely fresh Ubuntu installation?

Summary

I had to reinstall my NTB with Ubuntu 22.04 to have also GUI Desktop (from Ubuntu Server) I try to reproduce quickstart guide for running Kubeflow on my Laptop exactly as mentioned here https://charmed-kubeflow.io/docs/quickstart. All steps was done correctly until the following one:

aimaster@aimaster001:~$ juju bootstrap microk8s ERROR resolving microk8s credentials: ensuring cluster role “juju-credential-microk8s” in namespace “kube-system”: Get https://172.16.32.232:16443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-microk8s": Forbidden

What Should Happen Instead?

Instead it should deploy a Juju controller to the Kubernetes we set up with MicroK8s

Reproduction Steps

Any idea what am I doing wrong? Here is my setup: I am running following setup: Asus Zenbook Pro UX582LR, X86_64 Architecture, Ubuntu 22.04 LTS

Info:

juju 2.9.35 juju-bundle 0.4.0 juju-wait 2.8.4 kubectl 1.25.3 microk8s v1.24.6 aimaster@aimaster001:~$ microk8s inspect Inspecting system Inspecting Certificates Inspecting services Service snap.microk8s.daemon-cluster-agent is running Service snap.microk8s.daemon-containerd is running Service snap.microk8s.daemon-kubelite is running Service snap.microk8s.daemon-k8s-dqlite is running Service snap.microk8s.daemon-apiserver-kicker is running Copy service arguments to the final report tarball Inspecting AppArmor configuration Gathering system information Copy processes list to the final report tarball Copy snap list to the final report tarball Copy VM name (or none) to the final report tarball Copy disk usage information to the final report tarball Copy memory usage information to the final report tarball Copy server uptime to the final report tarball Copy current linux distribution to the final report tarball Copy openSSL information to the final report tarball Copy network configuration to the final report tarball Inspecting kubernetes cluster Inspect kubernetes cluster Inspecting dqlite Inspect dqlite

Building the report tarball Report tarball is at /var/snap/microk8s/4023/inspection-report- 20221102_183658.tar.gz aimaster@aimaster001:~$

aimaster@aimaster001:~$ microk8s status microk8s is running high-availability: no datastore master nodes: 127.0.0.1:19001 datastore standby nodes: none addons: enabled: dashboard # (core) The Kubernetes dashboard dns # (core) CoreDNS ha-cluster # (core) Configure high availability on the current node helm3 # (core) Helm 3 - Kubernetes package manager hostpath-storage # (core) Storage class; allocates storage from host directory ingress # (core) Ingress controller for external access metallb # (core) Loadbalancer for your Kubernetes cluster metrics-server # (core) K8s Metrics Server for API access to service metrics storage # (core) Alias to hostpath-storage add-on, deprecated disabled: community # (core) The community addons repository gpu # (core) Automatic enablement of Nvidia CUDA helm # (core) Helm 2 - the package manager for Kubernetes host-access # (core) Allow Pods connecting to Host services smoothly mayastor # (core) OpenEBS MayaStor prometheus # (core) Prometheus operator for monitoring and logging rbac # (core) Role-Based Access Control for authorisation registry # (core) Private image registry exposed on localhost:32000

Introspection Report

microk8s.inspect report inspection-report-20221103_130702.tar.gz

[ktsakalozos]

Hi @dominikkubik there seems to be a firewall running in your system. Is it possible you do a test run with it disabled so we can see if it standing in the way? If this is the case you should consider allowing traffic to the K8s services found in https://microk8s.io/docs/services-and-ports .

[dominikkubik]

Hello @ktsakalozos I tried what you suggested, but firewall seems not to be blocker in my case as you can see below.

aimaster@aimaster001:$ sudo ufw disable Firewall stopped and disabled on system startup aimaster@aimaster001:$ sudo ufw status Status: inactive aimaster@aimaster001:$ juju bootstrap microk8s ERROR resolving microk8s credentials: ensuring cluster role "juju-credential-microk8s" in namespace "kube-system": Get "https://172.16.32.232:16443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-microk8s": Forbidden aimaster@aimaster001:$ sudo juju bootstrap microk8s ERROR error reading current controller: cannot acquire lock file to get the current controller name: unable to open /tmp/juju-store-lock-3635383939333230: permission denied aimaster@aimaster001:$

Thank you very much for your time and help.

[neoaggelos]

Hi @dominikkubik

We have seen this Forbidden error popping up when an http proxy is configured on the system. Is this the case for you? If not, can you do the same steps with:

juju bootstrap microk8s --debug

to get some more verbose output? Are you having any issues with using MicroK8s in general? E.g. do microk8s kubectl get pod,node etc work properly?

[dominikkubik]

Hello @neoaggelos

Yes this is my case. My computer is part of network with proxy. I tried to set up Charmed Kubeflow ax described in this tutorial - https://charmed-kubeflow.io/tutorials/deploy-kubeflow-on-microk8s-behind-a-web-proxy#4-install-microk8s

I tried juju bootstrap microk8s --debug

and get following

aimaster@aimaster001:~$ juju bootstrap microk8s --debug 20:27:27 INFO juju.cmd supercommand.go:56 running juju [2.9.37 51672c0e4243f0d0e73f13cf1bbf5c5a9a720632 gc go1.18.8] 20:27:27 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/21315/bin/juju", "bootstrap", "microk8s", "--debug"} ERROR resolving microk8s credentials: ensuring cluster role "juju-credential-microk8s" in namespace "kube-system": Get "https://172.16.32.232:16443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-microk8s": Forbidden 20:27:28 DEBUG cmd supercommand.go:537 error stack: Get "https://172.16.32.232:16443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-microk8s": Forbidden github.com/juju/juju/caas/kubernetes/clientconfig.getOrCreateClusterRole:167: github.com/juju/juju/caas/kubernetes/clientconfig.ensureJujuAdminServiceAccount:88: ensuring cluster role "juju-credential-microk8s" in namespace "kube-system" github.com/juju/juju/caas/kubernetes/provider.attemptMicroK8sCredential:61: resolving microk8s credentials github.com/juju/juju/caas/kubernetes/provider.environProviderCredentials.RegisterCredentials:78: github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).cloud:1140: github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:623:

With Microk8s i tried what you suggested, and this was result:

aimaster@aimaster001:~$ microk8s kubectl get pod No resources found in default namespace. aimaster@aimaster001:~$ microk8s kubectl get node NAME STATUS ROLES AGE VERSION aimaster001 Ready <none> 2d6h v1.22.15-3+0434b23e1d7b0b

Thank you very much for your time and help.

[neoaggelos]

Can you check whether 172.16.0.0/12 is included in the list of the no_proxy variable?

[dominikkubik]

Hello @neoaggelos

I checked my configuration in /etc/environment and i had defined there 172.16.0.0/16. I change it to 172.16.0.0/12 and restarted machine. But now i get still same error as I pasted above in debug mode

[neoaggelos]

Hi @dominikkubik, I seem to have missed this, apologies.

In general, you have to ensure that the API server traffic is excluded from the http proxy. You could use something like tcpdump to see whether the traffic goes to the API server directly or your proxy.

For the most part, error messages like Get "https://172.16.32.232:16443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-microk8s": Forbidden indicate that the traffic is in fact still going through the proxy.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.