can't verify Kubernetes version: strconv.Atoi: parsing "": invalid syntax

mmack commented 6 years ago

Hey guys, please have a look at this problem:

`➜ kubernetes git:(master) ✗ kubeapps version Kubeapps Installer version: v0.2.0

➜ kubernetes git:(master) ✗ kubeapps up ERRO[0000] can't verify Kubernetes version: strconv.Atoi: parsing "": invalid syntax

➜ kubernetes git:(master) ✗ kubectl version Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0+coreos.0", GitCommit:"1b69a2a6c01194421b0aa17747a8c1a81738a8dd", GitTreeState:"clean", BuildDate:"2017-12-19T02:52:15Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} `

kubectl is working, cluster is online and working. What am i missing here?

Thanks, Max

prydonius commented 6 years ago

Thanks @mmack, this looks like it's coming from https://github.com/ksonnet/kubecfg/blob/master/utils/meta.go#L22. I'm not sure why it's failing though.

cc @anguslees @ngtuna

mmack commented 6 years ago

Any news on this? I wonder if something with my setup is wrong...

prydonius commented 6 years ago

@mmack I just noticed that, for whatever reason, the Major and Minor fields of the version.Info struct are empty: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0+coreos.0".... Since the function I linked to above is trying to parse this as an int, it's failing.

As far as I understand, we only need to use the version to check if your cluster is > 1.7 (https://github.com/kubeapps/kubeapps/blob/master/cmd/up.go#L91). Instead of failing because we can't verify the Kubernetes version, it might be better to silently continue.

I'm not sure why your cluster isn't returning the correct Major and Minor versions in those fields. Could be a bug with the CoreOS k8s distribution, the docs here (https://coreos.com/matchbox/docs/latest/bootkube-upgrades.html) show the Major and Minor versions correctly populated, albeit for an older version.

mmack commented 6 years ago

https://github.com/kubernetes/kubernetes/pull/57420

ngtuna commented 6 years ago

So what should we do in this issue from kubeapps's aspect ? Yes that check is only used for detecting 1.7+ k8s in order to apply RBAC and CRD. I don't think we should silently continue but maybe we need to catch more exceptions in the returned k8s version and make assumption with these cases. I will let you guys decide @mmack @prydonius

mmack commented 6 years ago

Imho would just show a warning that k8n version cannot be detected

prydonius commented 6 years ago

We could add a --force option to ignore the version check?

mmack commented 6 years ago

This problem should not exist anymore in a few days / weeks cause they fixed it in k8n, so a command line flag is overkill or not? :)

prydonius commented 6 years ago

Thanks for the info @mmack, yeah then I agree it's just better waiting. If the issue comes up again/is seen in other distros, then we can consider adding the flag.

ngtuna commented 6 years ago

@mmack I guess it has been fixed already. I don't have coreos k8s cluster to check. Can you @mmack ?

sriram18981 commented 6 years ago

Hi, I am running into the same problem. Please see below. I am using minikube and kubernetes version is 1.9.0

sriram@sriram-vaio ~/Downloads/linux-amd64 $ sudo curl -L https://github.com/kubeapps/installer/releases/download/v0.2.0/kubeapps-linux-amd64 -o /usr/local/bin/kubeapps && sudo chmod +x /usr/local/bin/kubeapps % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 148 0 148 0 0 82 0 --:--:-- 0:00:01 --:--:-- 82 100 611 0 611 0 0 291 0 --:--:-- 0:00:02 --:--:-- 596k 100 38.5M 100 38.5M 0 0 1859k 0 0:00:21 0:00:21 --:--:-- 2733k sriram@sriram-vaio ~/Downloads/linux-amd64 $ kubeapps up ERRO[0000] can't verify Kubernetes version: strconv.Atoi: parsing "": invalid syntax sriram@sriram-vaio ~/Downloads/linux-amd64 $ minikube stop Stopping local Kubernetes cluster... Machine stopped. sriram@sriram-vaio ~/Downloads/linux-amd64 $ kubeapps up ERRO[0018] can't verify Kubernetes version: Get https://192.168.99.100:8443/version: dial tcp 192.168.99.100:8443: getsockopt: no route to host sriram@sriram-vaio ~/Downloads/linux-amd64 $ minikube status minikube: Stopped cluster: kubectl: sriram@sriram-vaio ~/Downloads/linux-amd64 $ minikube start Starting local Kubernetes v1.9.0 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components... Kubectl is now configured to use the cluster. Loading cached images from config file. sriram@sriram-vaio ~/Downloads/linux-amd64 $ kubeapps up ERRO[0000] can't verify Kubernetes version: strconv.Atoi: parsing "": invalid syntax

sriram18981 commented 6 years ago

I just noticed something. from browser, if I go to the below URL to check for the version, it is returning authorization problems, and no version is mentioned in the output. Get https://192.168.99.100:8443/version

{ "kind": "Status", "apiVersion": "v1", "metadata": {

}, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 }

prydonius commented 6 years ago

@sriram18981 This has been fixed in a 1.9 patch release, but Minikube sticks to 1.9.0. Unfortunately I don't know if there is a way to use the latest 1.9 release with Minikube. I do know that you can run 1.8.0 and Kubeapps will work on that version (minikube start --kubernetes-version=v1.8.0).

cmarquezrusso commented 6 years ago

@prydonius There is an issue between minikube 0.25 and kubernetes 1.8.0: Kubernetes dashboard is not compatible --> https://github.com/kubernetes/minikube/issues/2521

@mmack How did you create the minikube RBAC machine? I tried:

minikube start --memory 8096 --extra-config=apiserver.Authorization.Mode=RBAC --kubernetes-version=v1.9.0

but I got issues with both, the DNS service and the dashboard. Update: This issue is filed here: https://github.com/kubernetes/minikube/issues/2510

More info here: https://asciinema.org/a/163965

Update: It seems possible to run 1.9.3 by changing the bootstrapper...

minikube start --bootstrapper kubeadm --kubernetes-version v1.9.3 --memory 4096 --extra-config=apiserver.Authorization.Mode=RBAC

^ This didnt work neither 😢 It's similar to this defect: https://github.com/kubernetes/minikube/issues/2040

migmartri commented 6 years ago

I can confirm this issue with Minikube,

My take on this (please correct me if I am wrong) is that this check could have been run in a more optimistic manner. Meaning that if we can detect the version, we handle it as expected, but if we can not, we default to latest (with CRD). At the end of the day, worst case scenario, kubeapps will not work for some cases instead of failing 100% of the cases.

The following sentence can be found in the readme, is <1.7 supported?

Kubeapps assumes a working Kubernetes (v1.7+) with RBAC enabled and kubectl installed and configured to talk to your Kubernetes cluster. Kubeapps binaries are available for both Linux and OS X, and Kubeapps has been tested with both minikube and Google Kubernetes Engine (GKE).

prydonius commented 6 years ago

@migmartri I actually agree, I think we should warn the user if we can't detect the version and continue. That said, this issue is specific to 1.9.0, it's a Kubernetes bug and has been fixed in patch versions.

Kubeapps does not support < 1.7. Following the 0.3.0 release of Kubeapps, we will only support 1.8+ (internally we require the CronJob API). Our goal is to support N and N-1 (so currently that would be 1.8 and 1.9) where N is the default version on most Kubernetes managed solutions (i.e. GKE, AKS, etc.)

cmarquezrusso commented 6 years ago

I found a workaround for kubeapps working on minikube over k8s 1.7.0... I hope you can use it.

Environment info:

minikube version: v0.23.0
kubectl version: v1.9.3
Kubeapps Installer version: v0.2.0

Start minikube with this (running k8s 1.7.0) minikube start --memory 4096 --kubernetes-version=v1.7.0 --extra-config=apiserver.Authorization.Mode=RBAC

Add this role to stop the issues with the dashboard and dns containers...

kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

Start kubeapps

~  kubeapps up                                                                                                                                               791ms  Wed 21 Feb 00:15:18 2018
INFO[0000] Updating namespaces kubeapps
INFO[0000]  Creating non-existent namespaces kubeapps
INFO[0000] Updating namespaces kubeless
INFO[0000]  Creating non-existent namespaces kubeless
INFO[0000] Updating configmaps kubeapps.tcp-services
[...]
You can run `kubectl get all --all-namespaces -l created-by=kubeapps` to check the status of the Kubeapps components.

Result:

 ~  kubectl get pods --all-namespaces                                                                                                                         101ms  Wed 21 Feb 00:27:33 2018
NAMESPACE     NAME                                        READY     STATUS    RESTARTS   AGE
kube-system   kube-addon-manager-minikube                 1/1       Running   0          14m
kube-system   kube-dns-1326421443-p1zmb                   3/3       Running   1          14m
kube-system   kubernetes-dashboard-0nk4h                  1/1       Running   3          14m
kube-system   sealed-secrets-controller-665256659-j4tsl   1/1       Running   0          12m
kubeapps      default-http-backend-3188807970-js9vx       1/1       Running   0          12m
kubeapps      kubeapps-dashboard-api-2565478011-m7pcq     2/2       Running   1          12m
kubeapps      kubeapps-dashboard-ui-3956904605-pmmv3      1/1       Running   0          12m
kubeapps      kubeless-ui-184132610-kdsvg                 2/2       Running   0          12m
kubeapps      mongodb-1543949793-4vs8j                    1/1       Running   0          12m
kubeapps      nginx-ingress-controller-3637265151-h5bvh   1/1       Running   0          12m
kubeless      kafka-0                                     1/1       Running   2          12m
kubeless      kubeless-controller-151920402-gwlq4         1/1       Running   0          12m
kubeless      zoo-0                                       1/1       Running   0          12m
 ~ 

prydonius commented 6 years ago

@cristian04 thanks for sharing the workaround! Does 1.8.0 work out of curiosity? Looks like the main issue is the DNS failure and the need to give kube-system cluster-admin access which has been reported here https://github.com/kubernetes/minikube/issues/1734/

cmarquezrusso commented 6 years ago

I didnt try it as per kubernetes/minikube#2521

weber-d commented 6 years ago

@cristian04 Not working for me. I followed your steps and still get the error

[user@kube ~]$ kubeapps up
ERRO[0000] can't verify Kubernetes version: strconv.Atoi: parsing "": invalid syntax

My server versions are empt too, which seems the cause of this issuse:

[user@kube ~]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2018-01-26T19:04:38Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

A fix is already provided here but not integrated in the binary. So it seems that we need to build the source on ourself to apply the fix.

Workaround: Build minikube from source

I could fix the issue by building minikube from source. The following describes how I did this.

Prerequisite: Install go

Since I haven't go installed, I need to install it first since minikube is written in go:

cd /tmp
curl -LO https://dl.google.com/go/go1.10.linux-amd64.tar.gz
sudo tar -C /usr/local -xvzf go1.10.linux-amd64.tar.gz
mkdir -p ~/projects/{bin,pkg,src}

# Make go avaliable in our path and set important variables
# See https://www.digitalocean.com/community/tutorials/how-to-install-go-1-7-on-centos-7
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bash_profile
echo 'export GOROOT=/usr/local/go' >> ~/.bash_profile
echo 'export GOBIN="$HOME/projects/bin"' >> ~/.bash_profile
echo 'export GOPATH="$HOME/projects/src"' >> ~/.bash_profile
source ~/.bash_profile

Now verify that go works properly:

[user@kube minikube]$ go version
go version go1.10 linux/amd64

Dependency: We need libvirt

See https://github.com/kubernetes/minikube/issues/2116#issuecomment-339195992 since I'm on CentOS7, this is the command to install the lib:

sudo yum install libvirt-devel.x86_64

Clone and Build go

# https://github.com/kubernetes/minikube/blob/master/docs/contributors/build_guide.md#building-from-source
git clone https://github.com/kubernetes/minikube.git $GOPATH/src/k8s.io/minikube
cd $GOPATH/src/k8s.io/minikube

After a few minutes, the command should exit without error and we've a binary in the dist folder:

[user@kube minikube]$ ls -lh out/minikube
-rwxrwxr-x. 1 user user 42M  5. Mär 14:39 out/minikube

Restart minikube

cd out
./minikube start

Now you should see correct versions on the server too:

[user@kube out]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}