On starting minikube, I am notified of the container having trouble accessing https://k8s.gcr.io, and the nodes not being ready.

What Happened?

When I run the command minikube start --nodes=2 I get the following output in the terminal:

 minikube start --nodes=2
😄  minikube v1.25.2 on Microsoft Windows 10 Enterprise 10.0.19043 Build 19043
✨  Automatically selected the docker driver. Other choices: hyperv, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ kubelet.cni-conf-dir=/etc/cni/net.mk
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

As a result, when I run kubectl get nodes, I get that the nodes I have just created have status NotReady.

Upon running kubectl describe nodes, I get:

Name:               minikube
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=minikube
                    kubernetes.io/os=linux
                    minikube.k8s.io/commit=362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7
                    minikube.k8s.io/name=minikube
                    minikube.k8s.io/primary=true
                    minikube.k8s.io/updated_at=2022_06_15T15_24_08_0700
                    minikube.k8s.io/version=v1.25.2
                    node-role.kubernetes.io/control-plane=
                    node-role.kubernetes.io/master=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 15 Jun 2022 15:24:03 +0200
Taints:             node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  minikube
  AcquireTime:     <unset>
  RenewTime:       Wed, 15 Jun 2022 15:26:19 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason
    Message
  ----             ------  -----------------                 ------------------                ------
    -------
  MemoryPressure   False   Wed, 15 Jun 2022 15:24:17 +0200   Wed, 15 Jun 2022 15:24:01 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 15 Jun 2022 15:24:17 +0200   Wed, 15 Jun 2022 15:24:01 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 15 Jun 2022 15:24:17 +0200   Wed, 15 Jun 2022 15:24:01 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Wed, 15 Jun 2022 15:24:17 +0200   Wed, 15 Jun 2022 15:24:01 +0200   KubeletNotReady
    container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  192.168.49.2
  Hostname:    minikube
Capacity:
  cpu:                8
  ephemeral-storage:  263174212Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             12747820Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  263174212Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             12747820Ki
  pods:               110
System Info:
  Machine ID:                 b6a262faae404a5db719705fd34b5c8b
  System UUID:                b6a262faae404a5db719705fd34b5c8b
  Boot ID:                    1cca22c5-5b6b-4efc-bfc7-3760907ff362
  Kernel Version:             5.10.102.1-microsoft-standard-WSL2
  OS Image:                   Ubuntu 20.04.2 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.12
  Kubelet Version:            v1.23.3
  Kube-Proxy Version:         v1.23.3
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
Non-terminated Pods:          (6 in total)
  Namespace                   Name                                CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                ------------  ----------  ---------------  -------------  ---
  kube-system                 etcd-minikube                       100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         2m20s
  kube-system                 kindnet-78znd                       100m (1%)     100m (1%)   50Mi (0%)        50Mi (0%)      2m10s
  kube-system                 kube-apiserver-minikube             250m (3%)     0 (0%)      0 (0%)           0 (0%)         2m20s
  kube-system                 kube-controller-manager-minikube    200m (2%)     0 (0%)      0 (0%)           0 (0%)         2m20s
  kube-system                 kube-proxy-hqflw                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2m10s
  kube-system                 kube-scheduler-minikube             100m (1%)     0 (0%)      0 (0%)           0 (0%)         2m20s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                750m (9%)   100m (1%)
  memory             150Mi (1%)  50Mi (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type    Reason                   Age    From             Message
  ----    ------                   ----   ----             -------
  Normal  Starting                 2m6s   kube-proxy
  Normal  Starting                 2m21s  kubelet          Starting kubelet.
  Normal  NodeHasSufficientMemory  2m21s  kubelet          Node minikube status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    2m21s  kubelet          Node minikube status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     2m21s  kubelet          Node minikube status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  2m21s  kubelet          Updated Node Allocatable limit across pods
  Normal  RegisteredNode           2m10s  node-controller  Node minikube event: Registered Node minikube in Controller

Name:               minikube-m02
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=minikube-m02
                    kubernetes.io/os=linux
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 15 Jun 2022 15:24:49 +0200
Taints:             node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  minikube-m02
  AcquireTime:     <unset>
  RenewTime:       Wed, 15 Jun 2022 15:26:20 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason
    Message
  ----             ------  -----------------                 ------------------                ------
    -------
  MemoryPressure   False   Wed, 15 Jun 2022 15:24:59 +0200   Wed, 15 Jun 2022 15:24:49 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 15 Jun 2022 15:24:59 +0200   Wed, 15 Jun 2022 15:24:49 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 15 Jun 2022 15:24:59 +0200   Wed, 15 Jun 2022 15:24:49 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Wed, 15 Jun 2022 15:24:59 +0200   Wed, 15 Jun 2022 15:24:49 +0200   KubeletNotReady
    container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  192.168.49.3
  Hostname:    minikube-m02
Capacity:
  cpu:                8
  ephemeral-storage:  263174212Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             12747820Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  263174212Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             12747820Ki
  pods:               110
System Info:
  Machine ID:                 b6a262faae404a5db719705fd34b5c8b
  System UUID:                b6a262faae404a5db719705fd34b5c8b
  Boot ID:                    1cca22c5-5b6b-4efc-bfc7-3760907ff362
  Kernel Version:             5.10.102.1-microsoft-standard-WSL2
  OS Image:                   Ubuntu 20.04.2 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.12
  Kubelet Version:            v1.23.3
  Kube-Proxy Version:         v1.23.3
PodCIDR:                      10.244.1.0/24
PodCIDRs:                     10.244.1.0/24
Non-terminated Pods:          (2 in total)
  Namespace                   Name                CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                ------------  ----------  ---------------  -------------  ---
  kube-system                 kindnet-fw894       100m (1%)     100m (1%)   50Mi (0%)        50Mi (0%)      99s
  kube-system                 kube-proxy-65pwx    0 (0%)        0 (0%)      0 (0%)           0 (0%)         99s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (1%)  100m (1%)
  memory             50Mi (0%)  50Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:
  Type    Reason                   Age                From             Message
  ----    ------                   ----               ----             -------
  Normal  Starting                 96s                kube-proxy
  Normal  Starting                 99s                kubelet          Starting kubelet.
  Normal  NodeHasSufficientMemory  99s (x2 over 99s)  kubelet          Node minikube-m02 status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    99s (x2 over 99s)  kubelet          Node minikube-m02 status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     99s (x2 over 99s)  kubelet          Node minikube-m02 status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  99s                kubelet          Updated Node Allocatable limit across pods
  Normal  RegisteredNode           95s                node-controller  Node minikube-m02 event: Registered Node minikube-m02 in Controller

As far as I can tell, the clue we need is in the output of the output above: for both nodes it says that Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

I have tried install calico and flannel as suggested here but flannel doesnt even complete the installation, whereas calico does finish, but doesnt resolve the problem upon deleting the current cluster and starting a new one.

Maybe relevant:

I do have WSL2 installed which is sharing the Docker installation (I installed Docker through Windows itself using Docker for Windows tho)
I am making use of a VPN my employer has installed on this pc.

Attach the log file

log.txt

Operating System

Windows

Driver

Docker

Hi @Wnzebkhan, thanks for reporting your issue with minikube.

We have a section on our website of how to use minikube with a VPN, could you take a look at that and see if the steps resolve your issue

https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/#vpn

@spowelljr; I have tried the following things mentioned in docs you linked:

Turn off the VPN => I tried this, but to no avail.
If you have access, whitelist the above IP ranges in your VPN software => Cannot add these, will have to check who I need to ask to have these added. But since turning VPN off didnt help, I am guessing it has nothing to do with the VPN at all.
In your VPN software, select an option similar to “Allow local (LAN) access when using VPN” (Cisco VPN example) => See point 2.
You may have luck selecting alternate values to the --host-only-cidr and --service-cluster-ip-range flags. => I am not sure what alternate values I should assign for these flags.

I am guessing the reason it is not working is because of the fact that my Docker installation is using WSL2 as back-end and not Hyper-V or some other back-end. Could that be a possible reason?

Edit: I have also noticed that the status of the nodes is NotReady when I have 2 nodes in my cluster. When I start a cluster with 1 node, I still get ❗ This container is having trouble accessing https://k8s.gcr.io but the status of the node is Ready and upon running kubectl describe nodes I do not get the error KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized.

Update: So I just found this page which says:

A vanilla minikube installation (minikube start) does not support any NetworkPolicies, since the default CNI, Kindnet, does not support Network Policies, by design.

Furthermore:

However, minikube can support NetworkPolicies if a supported CNI, such as Calico, is installed. In addition, in this scenario both Kubernetes network policy and Calico network policy are supported.

Hence, I have first run the following command to 'install Calico': curl https://docs.projectcalico.org/manifests/calico-typha.yaml -o calico.yaml after which I deleted my old cluster with minikube delete and then ran minikube start --nodes=2 --cni calico.

This initiated a cluster with 2 nodes who had status Ready. However, the message ❗ This container is having trouble accessing https://k8s.gcr.io upon creating the cluster still appeared.

Apparently, the CNI issue and the ❗ This container is having trouble accessing https://k8s.gcr.io are two seperate issues.

Not sure if this is related but got a similar message when upgrading to v1.26 WSL2 on Windows16 via Docker

😄  minikube v1.26.0 on Ubuntu 20.04 (amd64)
🆕  Kubernetes 1.24.1 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.24.1
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.23.3 preload ...
    > preloaded-images-k8s-v18-v1...: 400.43 MiB / 400.43 MiB  100.00% 14.00 Mi
❗  Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 2.475355583s
💡  Restarting the docker service may improve performance.
🏃  Updating the running docker "minikube" container ...
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image kubernetesui/dashboard:v2.6.0
    ▪ Using image kubernetesui/metrics-scraper:v1.0.8
🌟  Enabled addons: default-storageclass, storage-provisioner, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Also, I'd appreciate if somebody could point me to the issue (if there's one) related to pulling external images. I'm getting a lot of ErrImgPull and as a temporal workaround I'm pulling the images with Docker and adding them to the minikube cluster via minikube image load MyImage while also changing my image-pull-policy to never.

I saw something about adding a proxy, I went to the docs but I have no idea what to add for the environment variables mentioned there in order to pull from either Docker or somewhere else (I don't even know which port to add to the proxy).

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Not sure if this is related but got a similar message when upgrading to v1.26 WSL2 on Windows16 via Docker
😄  minikube v1.26.0 on Ubuntu 20.04 (amd64)
🆕  Kubernetes 1.24.1 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.24.1
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.23.3 preload ...
    > preloaded-images-k8s-v18-v1...: 400.43 MiB / 400.43 MiB  100.00% 14.00 Mi
❗  Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 2.475355583s
💡  Restarting the docker service may improve performance.
🏃  Updating the running docker "minikube" container ...
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image kubernetesui/dashboard:v2.6.0
    ▪ Using image kubernetesui/metrics-scraper:v1.0.8
🌟  Enabled addons: default-storageclass, storage-provisioner, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Also, I'd appreciate if somebody could point me to the issue (if there's one) related to pulling external images. I'm getting a lot of ErrImgPull and as a temporal workaround I'm pulling the images with Docker and adding them to the minikube cluster via minikube image load MyImage while also changing my image-pull-policy to never.

I saw something about adding a proxy, I went to the docs but I have no idea what to add for the environment variables mentioned there in order to pull from either Docker or somewhere else (I don't even know which port to add to the proxy).

I have same image pull problems when running common image even like nginx. I run minikube driven by docker on windows 10. Sometimes it tries several times and fix itself but really annoying sometimes error for pulling.

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/minikube/issues/14342#issuecomment-1335429861): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

Still facing this issue with gcr.io/k8s-minikube/kicbase:v0.0.30 (running with the docker driver) in WSL2. Any solutions here? :(

/reopen

@Tuanm: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes/minikube/issues/14342#issuecomment-1445032361): >Still facing this issue with `gcr.io/k8s-minikube/kicbase:v0.0.30` (running with the docker driver) in WSL2. Any solutions here? :( > >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

kubernetes / minikube