timed out waiting to elevate kube-system RBAC privileges: Post: Service Unavailable

[kikobg]

Hello,

I am trying to start local cluster on Windows 10 using minikube( v0.28.2 ). Everything runs fine until the step Starting cluster components... where I stumbled on the following issue:

E0727 15:00:05.218387 15960 start.go:300] Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: creating clusterrolebinding: Post https://192.168.99.100:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: Service Unavailable

Do you know what might be the reason?

Best regards

[amsaha]

I am seeing the same issue on macOS High Sierra 10.13.6 (17G65).

[Karmatronics]

Same thing for me also on macOS High Sierra 10.13.6 :(

[bakunin95]

Same issue on macOS High Sierra 10.13.6

[dprasanthv]

Uninstall/delete all minikube related files 1.1. minikube delete 1.2. rm /usr/local/minikube 1.3. rm -rf ~/.minikube

reinstall minikube.

[tbachman]

Tried the above (although for 1.2, a minor change is needed -- it's in /usr/local/bin), and still failed. macOS High Sierra 10.13.4.

[kikobg]

Works ok on Ubuntu VM(16.04) though. I managed to start a cluster on the VM with enabled nested virtualization and VirtualBox 5.2 installed.

[tbachman]

Not sure if this will help others, but this problem went away after upgrading VirtualBox.

[dinigo]

Same here. Can you tell the latest stable/working version?

Removing config and uninstalling didn't make the trick

[tbachman]

FWIW, I upgraded to 5.2.16 r123759 (Qt5.6.3) on macOS. I can't remember what version I was running previously, but it was at least a year old, maybe as much as two.

[nilebox]

Updating Virtual Box to the latest version helped on macOS as well.

[tbachman]

Just to provide a bit more background -- I realized that my localhost couldn't communicate with the VM on the host-only network (vboxnet4 in my case), which I believe is why I was getting this error from minikube. I did some googling that led me to links describing problems with older versions of Virtual Box, which is why I tried the upgrade.

[bakunin95]

I updated VirtualBox from 5.2.12 to 5.2.16 and i still get the error, i also tried other drivers. The only thing that works for me right now is: minikube start --bootstrapper=localkube

but localkube is deprecated so i don't really like the solution.

[tbachman]

Just as an FYI, I also followed the steps from dprasanthv to delete the VM after upgrading (wanted to make sure there wasn't any issue with the VM itself). I doubt this will make a difference, but I thought I should mention it, as it's something else that I did.

[zieg8301]

I'm experiencing the same issues above, but on Sierra (10.12.6). Updated virtualbox to latest version, cleaned and reinstalled minikube. Only thing that works is using localkube as the bootstrapper.

[InTheCloudDan]

Also hitting this, none of the above suggestions are working for me. Localkube seems to start but kubectl commands hang still.

[tbachman]

I have no idea why this works for me, but it appears to. My minikube creates a new host only network (e.g. vboxnet) whenever I create a new minikube. I ended up deleting the minikube, as well as the vboxnet network (along with some others that were no longer valid). I've found that if I do this every time between runs, it works. That said, it might work just as well if I turn around 3 times while patting my belly between runs.

[tbachman]

I finally have an answer that makes sense (spoiler alert: source is between keyboard and chair). I hadn't been paying attention to the times when I was using my VPN client and when I wasn't. Sure enough, I started having troubles with minikube again, and realized that I also had my VPN client on. As soon as I went off my VPN, minikube worked.

[cavapoo2]

same issue on ubuntu 16.04. minikube v0.28.2, virtual box 5.2.16 r123759 (Qt5.6.1) (not using a VPN) lots of these type errors 27387 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]

[ningg]

Solution：

• close the VPN.

[polaroidkidd]

I'm seeing the same issue on Ubuntu 18.04 and I am not running a VPN client

[VictorYou]

i tried this and it worked for me

export no_proxy=$no_proxy,192.168.99.100 minikube delete minikube start

i am working with Ubuntu18.04, minikube v0.28.2

[Baitanik]

I am also having same error .. but I am using vm-driver=none.. so virtualbox upgrade is not possible .. Did you find any other solution. I am running in RHEL 7 .

[mahorad]

same problem hrere.

[jamiekt]

Same problems here trying to start minikube. I'm on High Sierra (10.13.6). Virtualbox is 5.2.18 r124319 (Qt5.6.3).

Tried: minikube start --bootstrapper=localkube instead which successfully started minikube however
kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
fails with

error: failed to discover supported resources: Get https://192.168.99.100:8443/apis/extensions/v1beta1: Forbidden

[mahorad]

In my case, the culprit was the little snitch. Edited the rules and it worked properly

[jamiekt]

In my case, the culprit was the little snitch. Edited the rules and it worked properly

Hi @mahorad . Can you elaborate on "the little snitch" please? Which rules did you edit, and how? TIA.

[jgrasser]

@jamiekt - I ran into this issue as well and, as many people have stated in this thread, there was some conflict between my vpn settings and the default network configured on minikube. A broadly applicable solution is to override the minikube configuration with a subnet that is not routed to your vpn provider.

Instructions for Mac.

The default network used by minikube is some 192.168.99.xx.

Check the default routes for preexisting 192.168.99 rule. netstat -rn | grep 192.168.99 | awk '{ $NF }'

If the output is anything other than vboxnet you will have issues.

So, let's override the network settings used by minikube. I am using a subnet not advertised by my VPN ( 192.168.57 ). You can verify this by looking at the output of netstat -rn.. There should be no rules for that subnet. minikube config set host-only-cidr 192.168.57.1/24

Now start minikube.. minikube start

By the magic of networking, you should now see an entry in your route table (netstat -rn) for your new subnet.

[smadhumohan]

@jamiekt thanks for help, it worked for me

[marchmallow]

Updating to latest minikube v0.29.0 and virtualbox 5.2.18 on High Sierra (10.13.6) worked for me, none of other documented workarounds in this or similar issues did. So far running with VPN connection off have not tried turning that back on.

As far as the problem was occurring, netstat -rn | grep 192.168.99 did not return anything while now I get:

192.168.99         link#18            UC              2        0 vboxnet
192.168.99.100     8:0:27:3:4:63      UHLWI           0      277 vboxnet    835

My start command is minikube start --kubernetes-version v1.10.7 (and using kubectl v1.10.7 too)

[corcoja]

Had the same issue and after trying the above suggestions with no luck, I did some digging myself. It turns out it was related to VirtualBox – removed the vboxnet0 from the VirtualBox's Host Network Manager and started minikube again. Everything was working as intended!

It might be not the solution for everyone, but thought it could help someone else.

P. S. I'm running minikube on macOS v10.13.6 with minikube start.

[lokendra08]

Updating Virtual Box to the latest version helped on macOS as well.

updating virtual box worked for me as well:) on macos high sierra(10.13.6).

[bbhenry]

Updating VirtualBox worked. Thanks.

[danehans]

I ran into this issue and removing vboxnet0 worked for me. Thanks @iuliancorcoja for sharing the workaround.

[pmichali]

I have OS X Mohave (10.14), minikube 0.30.0, and I get the same timeout error. I tried removing the vboxnet0 entry in VBox, but still see the sam issue. I did do a uninstall and reinstall of minikube (and ~/.minikube/ area), as I thought I had an old version, but I still see the issue.

I was connected into work, via VPN, so I disconnected and retried. I see the same failure.

Any suggestion?

[tbachman]

After disconnecting the VPN, you might need to do a down/up on the vboxnet interface - I think when the VPN is brought up, the route to that interface gets removed/over-ridden, and doesn't automagically come back after bringing down the VPN

[pmichali]

I tried down/up of vboxnet0, but it still was showing the same issue. I restarted my Mac, and, without connecting to VPN, brought up minikube. It came up OK, except dashboard was in a crash loop. I brought down minikube, connected to VPN, and then tried again and I got the error:

E1119 07:54:11.849626    7305 start.go:302] Error restarting cluster:  restarting kube-proxy: waiting for kube-proxy to be up for configmap update: timed out waiting for the condition

[tbachman]

I guess the ifdown/up doesn't restore the route. In my case, I could see that there was no route to the vboxnet after going off VPN. I thought that it was restored with the ifdown/up, but maybe I just restored the route manually.

Not sure about the crash loop on minikube. Sounds like a different issue?

[russelldc]

I, too, am on OS X Mojave, with minikube 0.30.0 and I'm getting the same timeout error. I don't see any conflicts when I run netstat -rn

[bradj]

Opening VirtualBox and killing the vnet like @iuliancorcoja suggested worked for me! None of the other solutions seemed to work.

[JunyiJ]

Same issue

[vhosakot]

I saw the same error when starting minikube v0.30.0 (that runs k8s v1.10.0) on MacOS Mojave 10.14.1 with VirtualBox 5.2.22:

$ minikube start
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E1201 00:55:03.176754   56344 start.go:297] Error starting cluster:  timed out waiting to elevate kube-system RBAC privileges: creating clusterrolebinding: Post https://192.168.99.100:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 192.168.99.100:8443: i/o timeout

Per @iuliancorcoja's comment here in https://github.com/kubernetes/minikube/issues/3022#issuecomment-425495721, deleting vboxnet0 in the VirtualBox GUI under Global Tools --> Host Network Manager resolved this issue for me. Thanks @iuliancorcoja!

[erthornt]

I have this same issue with minkube v.0.30.0 on MacOS 10.13.6 and VirtualBox 5.2.22 r126460 (Qt5.6.3). I have started from the top of this thread and tried every single suggested fix for this and nothing has worked. On VPN, off VPN, wiped everything and reinstalled minikube, deleting vboxnet0 and also changing to 192.168.57 as the default host cidr. Nothing shows up in the netstat commands for the cluster at all. Is there anything else that can be tried?

[tbachman]

Not sure if others will find this helpful, but the root of my issue was I had no communication with my virtualbox VM -- specifically I was missing a route to the vboxnet0 network, even though I had an interface with an IP on that subnet (the VPN client I was using disallowed split-tunnel operation, so it deleted this route, and when the VPN was torn down, the route wasn't getting restored).

[erthornt]

Thanks Thomas,

So how do you add the route back after the VPN gets torn down?

From: Thomas Bachman notifications@github.com Reply-To: kubernetes/minikube reply@reply.github.com Date: Monday, 10 December 2018 at 19:09 To: kubernetes/minikube minikube@noreply.github.com Cc: Erly Thornton erthornt@cisco.com, Comment comment@noreply.github.com Subject: Re: [kubernetes/minikube] timed out waiting to elevate kube-system RBAC privileges: Post: Service Unavailable (#3022)

Not sure if others will find this helpful, but the root of my issue was I had no communication with my virtualbox VM -- specifically I was missing a route to the vboxnet0 network, even though I had an interface with an IP on that subnet (the VPN client I was using disallowed split-tunnel operation, so it deleted this route, and when the VPN was torn down, the route wasn't getting restored).

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/kubernetes/minikube/issues/3022#issuecomment-445934586, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AfCLfYaOBhnbDSq3SaLT1Y6R67YtddDbks5u3rGAgaJpZM4VjZJk.

[tbachman]

I believe if you delete the vboxnet, when it gets created it should add the route back. That said, if you are having the same problem I did (i.e. something like VPN preventing the route from being added), then this won't work (you can also find plenty of hits on google for ways to add static routes on different OSs).

[JunyiJ]

I just downloaded the most recent version of minikube v0.31.0, MacOS 10.14, VirtualBox 5.2.22 r126460. The other change I made is set no_proxy: export NO_PROXY=$no_proxy,$(minikube ip). Now my problem is fixed.

[petermikitsh]

@iuliancorcoja's solution worked for me. Thanks for commenting!

[mkdavis]

Solution suggesting to remove vboxnet0 network entry in VirtualBox worked for me !! On MacOS Mojave.
I had to back off minikube v0.31.0 and use minikube v0.30.0. Also, using VirtualBox 6.0

[zhangxiaoyu-zidif]

I am seeing the same issue on macOS High Sierra 10.13.5

minikube start --feature-gates=AllAlpha=true --vm-driver=virtualbox
Starting local Kubernetes v1.12.4 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Starting cluster components...
E0102 16:52:23.568162    8681 start.go:343] Error starting cluster:  timed out waiting to elevate kube-system RBAC privileges: creating clusterrolebinding: Post https://192.168.99.103:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: EOF

[zhangxiaoyu-zidif]

export NO_PROXY=$no_proxy,$(minikube ip). My problem is fixed.