search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.35k stars 4.87k forks source link

Error downloading kic artifacts #19260

Open chrisHLY opened 3 months ago

chrisHLY commented 3 months ago

重现问题所需的命令： minikube start --memory=4096 --cpus=2 --kubernetes-version=v1.30.0

失败的命令的完整输出：

E0716 16:30:50.296015 18021 cache.go:189] Error downloading kic artifacts: failed to download kic base image or any fallback image 🔥 Creating docker container (CPUs=2, Memory=4096MB) ... 🤦 StartHost failed, but will try again: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -d /var/lib: exit status 125 stdout:

stderr: Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e' locally docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See 'docker run --help'.

🤷 docker "minikube" 缺失 container，将重新创建。 🔥 Creating docker container (CPUs=2, Memory=4096MB) ... 😿 启动 docker container 失败。运行 "minikube delete" 可能需要修复它： recreate: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -d /var/lib: exit status 125 stdout:

stderr: Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e' locally docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See 'docker run --help'.

❌ 因 GUEST_PROVISION 错误而退出：error provisioning guest: Failed to start host: recreate: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -d /var/lib: exit status 125 stdout:

stderr: Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e' locally docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See 'docker run --help'.

minikube logs命令的输出：

logs.txt

使用的操作系统版本： macOs Ventura 13.0.1

ComradeProgrammer commented 3 months ago

@medyagh @spowelljr I guess the reason is like this: at pkg/drivers/kic/kic.go:180, oci.PrepareContainerNode was called, before the extraction of preload tarball (which happens at pkg/drivers/kic/kic.go:187, 7 lines after oci.PrepareContainerNode was called)

Under this circumstance:

  1. oci.PrepareContainerNode will try to run prepareVolumeSideCar, which will start a kicbase container.
  2. But at this time the preload has not been extracted, so docker will try to pull image via gcr.io or dockerhub
  3. In China mainland, pulling kicbase via gcr.io or dockerhub will fail due to GFW, so the start process fails

Does this make sense?

BTW I think I was wrong about the reason why preload was not used in China mainland. It seems that it was not proxy. And this seems to be the actual reason why preload never works in China mainland

spowelljr commented 3 months ago

@ComradeProgrammer I don't think this is the issue. oci.PrepareContainerNode creates the minikube and minikube-preload-sidecar. Once that's done it extracts the preloads to the volume created in oci.PrepareContainerNode.

oci.PrepareContainerNode:

docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true

docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var docker.io/kicbase/build:v0.0.44-1721064868-19249@sha256:f2789f25c9e51cdeb9cef760e15dc838ef08abd5bb1913311c1eabedda231e8c -d /var/lib

oci.ExtractTarballToVolume:

docker run --rm --entrypoint /usr/bin/tar -v /Users/powellsteven/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.2-docker-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir docker.io/kicbase/build:v0.0.44-1721064868-19249@sha256:f2789f25c9e51cdeb9cef760e15dc838ef08abd5bb1913311c1eabedda231e8c -I lz4 -xf /preloaded.tar -C /extractDir

I assume you're referencing the log docker run --rm --name minikube-preload-sidecar, that's called in oci.PrepareContainerNode, not oci.ExtractTarballToVolume. In the logs above the user never even hits oci.ExtractTarballToVolume because because creating the volume with the kicbase fails first.

ComradeProgrammer commented 3 months ago

So accoding to the discussion on Wednesday

k8s-triage-robot commented 1 day ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale