Persistently failing CI job for k8s 1.13.x and minikube 1.5

[betatim]

One of the CI jobs consistently fails to start k8s on minikube.

It is kubernetes version 1.13.12 on minikube 1.5.2. We see the following error:

+sudo /home/travis/build/jupyterhub/kubespawner/bin/minikube start --vm-driver=none --kubernetes-version=v1.13.12
* minikube v1.5.2 on Ubuntu 18.04
  - MINIKUBE_WANTUPDATENOTIFICATION=false
  - KUBECONFIG=/home/travis/.kube/config
  - MINIKUBE_WANTREPORTERRORPROMPT=false
  - MINIKUBE_HOME=/home/travis
  - MINIKUBE_VERSION=1.5.2
* Running on localhost (CPUs=2, Memory=7974MB, Disk=69295MB) ...
* OS release is Ubuntu 18.04.3 LTS
* Preparing Kubernetes v1.13.12 on Docker '18.06.0-ce' ...
  - kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
* Downloading kubelet v1.13.12
* Downloading kubeadm v1.13.12
* Pulling images ...
* Launching Kubernetes ... 
* X Error starting cluster: init failed. cmd: "/bin/bash -c \"sudo env PATH=/var/lib/minikube/binaries/v1.13.12:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap\"": command failed: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.13.12:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap"

stdout: [init] Using Kubernetes version: v1.13.12
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.0.0-1026-gcp

Is this an incompatibility between the minikube and kubernetes version?

Build log for one of these jobs: https://travis-ci.org/jupyterhub/kubespawner/jobs/642254339?utm_medium=notification&utm_source=github_status

[consideRatio]

I really don't understand this yet, but here is my ramble of related confusion:

This is one of the reasons I started disliking minikube initially. This didn't fail initially, but then it did. I think it because the Travis CI Ubuntu system bumped their Linux kernel and minikube for k8s 1.13 decides to be angry about it.

I don't think we can specify the use of anything more specific than "bionic" or "xenial" etc.

[betatim]

I thought minikube brings its own VM to run in so that we are isolated from what the host does?

Next thing to do is find a list of (in)compatible versions of linux kernel, minikube and kubernetes.

Some software uses odd-even numbering for their releases. So for example 0.2 is a "good, stable, long term release", but 0.3 is more of a short lived experimental release which normal users shouldn't be using. Then 0.4 is a "good" release again. Is there something like that for kubernetes? Then we could say we don't test 1.13 because it was meant as a short lived "experimental" release.

[consideRatio]

@betatim I've seen no indication that kubernetes minor versions are differentiated between odd/even, but I have seen some statements about "the last X number of minor releases will be provided security fixes" etc.

Minikube can bring its own VM to run in, but using --vm-driver we say that it shouldn't start a VM.

https://github.com/jupyterhub/kubespawner/blob/892780ae86c6e74172065ab984bf2d059d00c31f/ci/install-kube.sh#L21-L22

[consideRatio]

Two suggestions that could be worth trying to resolve this issue:

1. We try to run the k8s 1.13 job with dist: xenial
2. We try to update minikube

[consideRatio]

I try those ideas here: https://github.com/jupyterhub/kubespawner/pull/394

[consideRatio]

Using xenial for k8s minikube starting a k8s 1.13 cluster did the trick!