Closed dctwan closed 3 years ago
andriyDev
commented
3 years ago Could you provide the full set of logs using minikube logs -f file.txt? We have a main issue here https://github.com/kubernetes/minikube/issues/10544. The current workaround is minikube delete --all --purge and restart, but I've seen several situations where this hasn't worked.
Bytesu
commented
3 years ago Could you provide the full set of logs using
minikube logs -f file.txt? We have a main issue here #10544. The current workaround isminikube delete --all --purgeand restart, but I've seen several situations where this hasn't worked.
| => Audit <== | --------- | ------ | ---------- | ------ | --------- | ------------------------------- | ------------------------------- | Command | Args | Profile | User | Version | Start Time | End Time | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| start | minikube | k8s | v1.22.0 | Sat, 21 Aug 2021 14:42:52 CST | Sat, 21 Aug 2021 14:48:44 CST | ||||||||||
| --------- | ------ | ---------- | ------ | --------- | ------------------------------- | ------------------------------- |
==> Last Start <==
Log file created at: 2021/08/21 14:42:52
Running on machine: iZ8vb1cgnytsf2bnokinqcZ
Binary: Built with gc go1.16.4 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0821 14:42:52.714001 3693658 out.go:286] Setting OutFile to fd 1 ...
I0821 14:42:52.714102 3693658 out.go:338] isatty.IsTerminal(1) = true
I0821 14:42:52.714105 3693658 out.go:299] Setting ErrFile to fd 2...
I0821 14:42:52.714110 3693658 out.go:338] isatty.IsTerminal(2) = true
I0821 14:42:52.714252 3693658 root.go:312] Updating PATH: /home/k8s/.minikube/bin
W0821 14:42:52.714384 3693658 root.go:291] Error reading config file at /home/k8s/.minikube/config/config.json: open /home/k8s/.minikube/config/config.json: no such file or directory
I0821 14:42:52.714604 3693658 out.go:293] Setting JSON to false
I0821 14:42:52.721184 3693658 start.go:111] hostinfo: {"hostname":"iZ8vb1cgnytsf2bnokinqcZ","uptime":1376177,"bootTime":1628151995,"procs":137,"os":"linux","platform":"centos","platformFamily":"rhel","platformVersion":"7.6.1810","kernelVersion":"3.10.0-957.5.1.el7.x86_64","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"guest","hostId":"20190215-1721-0859-0907-433256076310"}
I0821 14:42:52.721283 3693658 start.go:121] virtualization: guest
I0821 14:42:52.723904 3693658 out.go:165] 😄 minikube v1.22.0 on Centos 7.6.1810 (amd64)
I0821 14:42:52.724092 3693658 notify.go:169] Checking for updates...
I0821 14:42:52.724139 3693658 driver.go:335] Setting default libvirt URI to qemu:///system
I0821 14:42:52.724164 3693658 global.go:111] Querying for installed drivers using PATH=/home/k8s/.minikube/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
I0821 14:42:52.724191 3693658 global.go:119] podman default: true priority: 7, state: {Installed:false Healthy:false Running:false NeedsImprovement:false Error:exec: "podman": executable file not found in $PATH Reason: Fix:Install Podman Doc:https://minikube.sigs.k8s.io/docs/drivers/podman/}
I0821 14:42:52.724229 3693658 global.go:119] ssh default: false priority: 4, state: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:
stderr: Error: No such network: minikube I0821 14:48:01.920036 3693658 network_create.go:260] output of [docker network inspect minikube]: -- stdout -- []
-- /stdout -- stderr Error: No such network: minikube
/stderr
I0821 14:48:01.920089 3693658 cli_runner.go:115] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0821 14:48:01.957568 3693658 network.go:288] reserving subnet 192.168.49.0 for 1m0s: &{mu:{state:0 sema:0} read:{v:{m:map[] amended:true}} dirty:map[192.168.49.0:0xc0001c22c8] misses:0}
I0821 14:48:01.957605 3693658 network.go:235] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:}}
I0821 14:48:01.957625 3693658 network_create.go:106] attempt to create docker network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 1500 ...
I0821 14:48:01.957671 3693658 cli_runner.go:115] Run: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true minikube
I0821 14:48:02.034689 3693658 network_create.go:90] docker network minikube 192.168.49.0/24 created
I0821 14:48:02.034719 3693658 kic.go:106] calculated static IP "192.168.49.2" for the "minikube" container
I0821 14:48:02.034789 3693658 cli_runner.go:115] Run: docker ps -a --format {{.Names}}
I0821 14:48:02.073358 3693658 cli_runner.go:115] Run: docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0821 14:48:02.111105 3693658 oci.go:102] Successfully created a docker volume minikube
I0821 14:48:02.111170 3693658 cli_runner.go:115] Run: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var kicbase/stable:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 -d /var/lib
I0821 14:48:02.633510 3693658 oci.go:106] Successfully prepared a docker volume minikube
W0821 14:48:02.633570 3693658 oci.go:135] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
W0821 14:48:02.633584 3693658 oci.go:119] Your kernel does not support memory limit capabilities or the cgroup is not mounted.
I0821 14:48:02.633652 3693658 cli_runner.go:115] Run: docker info --format "'{{json .SecurityOptions}}'"
I0821 14:48:02.633776 3693658 preload.go:134] Checking if preload exists for k8s version v1.21.2 and runtime docker
I0821 14:48:02.633799 3693658 kic.go:179] Starting extracting preloaded images to volume ...
I0821 14:48:02.633850 3693658 cli_runner.go:115] Run: docker run --rm --entrypoint /usr/bin/tar -v /home/k8s/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v11-v1.21.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir kicbase/stable:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 -I lz4 -xf /preloaded.tar -C /extractDir
I0821 14:48:02.784613 3693658 cli_runner.go:115] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 kicbase/stable:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79
I0821 14:48:03.308138 3693658 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Running}}
I0821 14:48:03.368103 3693658 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0821 14:48:03.415844 3693658 cli_runner.go:115] Run: docker exec minikube stat /var/lib/dpkg/alternatives/iptables
I0821 14:48:03.525892 3693658 oci.go:278] the created container "minikube" has a running status.
I0821 14:48:03.525917 3693658 kic.go:210] Creating ssh key for kic: /home/k8s/.minikube/machines/minikube/id_rsa...
I0821 14:48:03.859062 3693658 kic_runner.go:188] docker (temp): /home/k8s/.minikube/machines/minikube/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0821 14:48:04.066187 3693658 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0821 14:48:04.120101 3693658 kic_runner.go:94] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0821 14:48:04.120118 3693658 kic_runner.go:115] Args: [docker exec --privileged minikube chown docker:docker /home/docker/.ssh/authorized_keys]
I0821 14:48:09.953763 3693658 cli_runner.go:168] Completed: docker run --rm --entrypoint /usr/bin/tar -v /home/k8s/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v11-v1.21.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir kicbase/stable:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 -I lz4 -xf /preloaded.tar -C /extractDir: (7.319871432s)
I0821 14:48:09.953790 3693658 kic.go:188] duration metric: took 7.319993 seconds to extract preloaded images to volume
I0821 14:48:09.953861 3693658 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0821 14:48:10.005386 3693658 machine.go:88] provisioning docker machine ...
I0821 14:48:10.005437 3693658 ubuntu.go:169] provisioning hostname "minikube"
I0821 14:48:10.005503 3693658 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0821 14:48:10.045254 3693658 main.go:130] libmachine: Using SSH client type: native
I0821 14:48:10.045451 3693658 main.go:130] libmachine: &{{{
I0821 14:48:10.173491 3693658 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0821 14:48:10.214897 3693658 main.go:130] libmachine: Using SSH client type: native
I0821 14:48:10.215162 3693658 main.go:130] libmachine: &{{{
if ! grep -xq '.*\sminikube' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 minikube/g' /etc/hosts;
else
echo '127.0.1.1 minikube' | sudo tee -a /etc/hosts;
fi
fiI0821 14:48:10.332798 3693658 main.go:130] libmachine: SSH cmd err, output:
I0821 14:48:10.979554 3693658 ubuntu.go:71] root file system type: overlay
I0821 14:48:10.979741 3693658 provision.go:308] Updating docker unit: /lib/systemd/system/docker.service ...
I0821 14:48:10.979798 3693658 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0821 14:48:11.021169 3693658 main.go:130] libmachine: Using SSH client type: native
I0821 14:48:11.021341 3693658 main.go:130] libmachine: &{{{
[Service] Type=notify Restart=on-failure
ExecStart= ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 ExecReload=/bin/kill -s HUP \$MAINPID
LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity
TasksMax=infinity TimeoutStartSec=0
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0821 14:48:11.147588 3693658 main.go:130] libmachine: SSH cmd err, output:
[Service] Type=notify Restart=on-failure
ExecStart= ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity
TasksMax=infinity TimeoutStartSec=0
Delegate=yes
KillMode=process
[Install] WantedBy=multi-user.target
I0821 14:48:11.147655 3693658 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0821 14:48:11.188262 3693658 main.go:130] libmachine: Using SSH client type: native
I0821 14:48:11.188448 3693658 main.go:130] libmachine: &{{{
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229. -# Both the old, and new location are accepted by systemd 229 and up, so using the old location -# to make them work for either version of systemd. -StartLimitBurst=3 +Restart=on-failure
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230. -# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make -# this option work for either version of systemd. -StartLimitInterval=60s + + +# This file is a systemd drop-in unit that inherits from the base dockerd configuration. +# The base configuration already specifies an 'ExecStart=...' command. The first directive +# here is to clear out that command inherited from the base configuration. Without this, +# the command from the base configuration and the command specified here are treated as +# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd +# will catch this invalid input and refuse to start the service with an error like: +# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. + +# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other +# container runtimes. If left unlimited, it may result in OOM issues with MySQL. +ExecStart= +ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 +ExecReload=/bin/kill -s HUP $MAINPID
@@ -32,16 +34,16 @@ LimitNPROC=infinity LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it. -# Only systemd 226 and above support this option. +# Uncomment TasksMax if your systemd version supports it. +# Only systemd 226 and above support this version. TasksMax=infinity +TimeoutStartSec=0
Delegate=yes
KillMode=process -OOMScoreAdjust=-500
[Install] WantedBy=multi-user.target Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable docker
I0821 14:48:12.028112 3693658 machine.go:91] provisioned docker machine in 2.022710435s I0821 14:48:12.028125 3693658 client.go:171] LocalClient.Create took 10.583434327s I0821 14:48:12.028138 3693658 start.go:168] duration metric: libmachine.API.Create for "minikube" took 10.583488049s I0821 14:48:12.028154 3693658 start.go:267] post-start starting for "minikube" (driver="docker") I0821 14:48:12.028161 3693658 start.go:277] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs] I0821 14:48:12.028229 3693658 ssh_runner.go:149] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs I0821 14:48:12.028286 3693658 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube I0821 14:48:12.070372 3693658 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49242 SSHKeyPath:/home/k8s/.minikube/machines/minikube/id_rsa Username:docker} I0821 14:48:12.155629 3693658 ssh_runner.go:149] Run: cat /etc/os-release I0821 14:48:12.158859 3693658 main.go:130] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found I0821 14:48:12.158873 3693658 main.go:130] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found I0821 14:48:12.158882 3693658 main.go:130] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found I0821 14:48:12.158905 3693658 info.go:137] Remote host: Ubuntu 20.04.2 LTS I0821 14:48:12.158918 3693658 filesync.go:126] Scanning /home/k8s/.minikube/addons for local assets ... I0821 14:48:12.158986 3693658 filesync.go:126] Scanning /home/k8s/.minikube/files for local assets ... I0821 14:48:12.159008 3693658 start.go:270] post-start completed in 130.847812ms I0821 14:48:12.159282 3693658 cli_runner.go:115] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube I0821 14:48:12.199153 3693658 profile.go:148] Saving config to /home/k8s/.minikube/profiles/minikube/config.json ... I0821 14:48:12.199395 3693658 ssh_runner.go:149] Run: sh -c "df -h /var | awk 'NR==2{print $5}'" I0821 14:48:12.199434 3693658 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube I0821 14:48:12.241619 3693658 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49242 SSHKeyPath:/home/k8s/.minikube/machines/minikube/id_rsa Username:docker} I0821 14:48:12.323106 3693658 start.go:129] duration metric: createHost completed in 10.882332931s I0821 14:48:12.323123 3693658 start.go:80] releasing machines lock for "minikube", held for 10.882493029s I0821 14:48:12.323205 3693658 cli_runner.go:115] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube I0821 14:48:12.363515 3693658 ssh_runner.go:149] Run: systemctl --version I0821 14:48:12.363539 3693658 ssh_runner.go:149] Run: curl -sS -m 2 https://k8s.gcr.io/ I0821 14:48:12.363565 3693658 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube I0821 14:48:12.363594 3693658 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube I0821 14:48:12.407945 3693658 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49242 SSHKeyPath:/home/k8s/.minikube/machines/minikube/id_rsa Username:docker} I0821 14:48:12.408282 3693658 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49242 SSHKeyPath:/home/k8s/.minikube/machines/minikube/id_rsa Username:docker} I0821 14:48:14.501768 3693658 ssh_runner.go:189] Completed: systemctl --version: (2.138226419s) I0821 14:48:14.501841 3693658 ssh_runner.go:189] Completed: curl -sS -m 2 https://k8s.gcr.io/: (2.138286827s) I0821 14:48:14.501853 3693658 ssh_runner.go:149] Run: sudo systemctl is-active --quiet service containerd W0821 14:48:14.501869 3693658 start.go:655] [curl -sS -m 2 https://k8s.gcr.io/] failed: curl -sS -m 2 https://k8s.gcr.io/: Process exited with status 28 stdout:
stderr: curl: (28) Connection timed out after 2000 milliseconds W0821 14:48:14.502083 3693658 out.go:230] ❗ This container is having trouble accessing https://k8s.gcr.io W0821 14:48:14.502159 3693658 out.go:230] 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ I0821 14:48:14.513538 3693658 ssh_runner.go:149] Run: sudo systemctl cat docker.service I0821 14:48:14.524209 3693658 cruntime.go:249] skipping containerd shutdown because we are bound to it I0821 14:48:14.524257 3693658 ssh_runner.go:149] Run: sudo systemctl is-active --quiet service crio I0821 14:48:14.534107 3693658 ssh_runner.go:149] Run: /bin/bash -c "sudo mkdir -p /etc && printf %!s(MISSING) "runtime-endpoint: unix:///var/run/dockershim.sock image-endpoint: unix:///var/run/dockershim.sock " | sudo tee /etc/crictl.yaml" I0821 14:48:14.551899 3693658 ssh_runner.go:149] Run: sudo systemctl unmask docker.service I0821 14:48:14.626612 3693658 ssh_runner.go:149] Run: sudo systemctl enable docker.socket I0821 14:48:14.698380 3693658 ssh_runner.go:149] Run: sudo systemctl cat docker.service I0821 14:48:14.708483 3693658 ssh_runner.go:149] Run: sudo systemctl daemon-reload I0821 14:48:14.776286 3693658 ssh_runner.go:149] Run: sudo systemctl start docker I0821 14:48:14.786443 3693658 ssh_runner.go:149] Run: docker version --format {{.Server.Version}} I0821 14:48:14.846170 3693658 ssh_runner.go:149] Run: docker version --format {{.Server.Version}} I0821 14:48:14.906887 3693658 out.go:192] 🐳 Preparing Kubernetes v1.21.2 on Docker 20.10.7 ... I0821 14:48:14.906982 3693658 cli_runner.go:115] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" I0821 14:48:14.945761 3693658 ssh_runner.go:149] Run: grep 192.168.49.1 host.minikube.internal$ /etc/hosts I0821 14:48:14.949681 3693658 ssh_runner.go:149] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.49.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts"" I0821 14:48:14.960138 3693658 preload.go:134] Checking if preload exists for k8s version v1.21.2 and runtime docker I0821 14:48:14.960183 3693658 ssh_runner.go:149] Run: docker images --format {{.Repository}}:{{.Tag}} I0821 14:48:15.010825 3693658 docker.go:535] Got preloaded images: -- stdout -- k8s.gcr.io/kube-apiserver:v1.21.2 k8s.gcr.io/kube-scheduler:v1.21.2 k8s.gcr.io/kube-proxy:v1.21.2 k8s.gcr.io/kube-controller-manager:v1.21.2 gcr.io/k8s-minikube/storage-provisioner:v5 k8s.gcr.io/pause:3.4.1 kubernetesui/dashboard:v2.1.0 k8s.gcr.io/coredns/coredns:v1.8.0 k8s.gcr.io/etcd:3.4.13-0 kubernetesui/metrics-scraper:v1.0.4
-- /stdout -- I0821 14:48:15.010843 3693658 docker.go:466] Images already preloaded, skipping extraction I0821 14:48:15.010889 3693658 ssh_runner.go:149] Run: docker images --format {{.Repository}}:{{.Tag}} I0821 14:48:15.060991 3693658 docker.go:535] Got preloaded images: -- stdout -- k8s.gcr.io/kube-apiserver:v1.21.2 k8s.gcr.io/kube-scheduler:v1.21.2 k8s.gcr.io/kube-proxy:v1.21.2 k8s.gcr.io/kube-controller-manager:v1.21.2 gcr.io/k8s-minikube/storage-provisioner:v5 k8s.gcr.io/pause:3.4.1 kubernetesui/dashboard:v2.1.0 k8s.gcr.io/coredns/coredns:v1.8.0 k8s.gcr.io/etcd:3.4.13-0 kubernetesui/metrics-scraper:v1.0.4
-- /stdout -- I0821 14:48:15.061022 3693658 cache_images.go:74] Images are preloaded, skipping loading I0821 14:48:15.061076 3693658 ssh_runner.go:149] Run: docker info --format {{.CgroupDriver}} I0821 14:48:15.165551 3693658 cni.go:93] Creating CNI manager for "" I0821 14:48:15.165565 3693658 cni.go:167] CNI unnecessary in this configuration, recommending no CNI I0821 14:48:15.165575 3693658 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16 I0821 14:48:15.165591 3693658 kubeadm.go:153] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.49.2 APIServerPort:8443 KubernetesVersion:v1.21.2 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:minikube NodeName:minikube DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.49.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NoTaintMaster:true NodeIP:192.168.49.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticPodPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[]} I0821 14:48:15.165738 3693658 kubeadm.go:157] kubeadm config: apiVersion: kubeadm.k8s.io/v1beta2 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.49.2 bindPort: 8443 bootstrapTokens:
apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration authentication: x509: clientCAFile: /var/lib/minikube/certs/ca.crt cgroupDriver: cgroupfs clusterDomain: "cluster.local"
apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration clusterCIDR: "10.244.0.0/16" metricsBindAddress: 0.0.0.0:10249 conntrack: maxPerCore: 0
I0821 14:48:15.165827 3693658 kubeadm.go:909] kubelet [Unit] Wants=docker.socket
[Service] ExecStart= ExecStart=/var/lib/minikube/binaries/v1.21.2/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.49.2
[Install]
config:
{KubernetesVersion:v1.21.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0821 14:48:15.165880 3693658 ssh_runner.go:149] Run: sudo ls /var/lib/minikube/binaries/v1.21.2
I0821 14:48:15.174204 3693658 binaries.go:44] Found k8s binaries, skipping transfer
I0821 14:48:15.174261 3693658 ssh_runner.go:149] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0821 14:48:15.182364 3693658 ssh_runner.go:316] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (334 bytes)
I0821 14:48:15.196199 3693658 ssh_runner.go:316] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0821 14:48:15.209784 3693658 ssh_runner.go:316] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (1867 bytes)
I0821 14:48:15.223088 3693658 ssh_runner.go:149] Run: grep 192.168.49.2 control-plane.minikube.internal$ /etc/hosts
I0821 14:48:15.226107 3693658 ssh_runner.go:149] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.49.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0821 14:48:15.235541 3693658 certs.go:52] Setting up /home/k8s/.minikube/profiles/minikube for IP: 192.168.49.2
I0821 14:48:15.235569 3693658 certs.go:183] generating minikubeCA CA: /home/k8s/.minikube/ca.key
I0821 14:48:15.397127 3693658 crypto.go:157] Writing cert to /home/k8s/.minikube/ca.crt ...
I0821 14:48:15.397144 3693658 lock.go:36] WriteFile acquiring /home/k8s/.minikube/ca.crt: {Name:mk78049a0e4179336a027921a9db7ade040e6a7b Clock:{} Delay:500ms Timeout:1m0s Cancel:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0821 14:48:16.262523 3693658 ssh_runner.go:240] Start: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.21.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0821 14:48:17.192597 3693658 out.go:192] ▪ Generating certificates and keys ...
I0821 14:48:20.427081 3693658 out.go:192] ▪ Booting up control plane ...
I0821 14:48:36.490053 3693658 out.go:192] ▪ Configuring RBAC rules ...
I0821 14:48:36.907229 3693658 cni.go:93] Creating CNI manager for ""
I0821 14:48:36.907244 3693658 cni.go:167] CNI unnecessary in this configuration, recommending no CNI
I0821 14:48:36.907287 3693658 ssh_runner.go:149] Run: /bin/bash -c "cat /proc/$(pgrep kube-apiserver)/oom_adj"
I0821 14:48:36.907423 3693658 ssh_runner.go:149] Run: sudo /var/lib/minikube/binaries/v1.21.2/kubectl create clusterrolebinding minikube-rbac --clusterrole=cluster-admin --serviceaccount=kube-system:default --kubeconfig=/var/lib/minikube/kubeconfig
I0821 14:48:36.907489 3693658 ssh_runner.go:149] Run: sudo /var/lib/minikube/binaries/v1.21.2/kubectl label nodes minikube.k8s.io/version=v1.22.0 minikube.k8s.io/commit=a03fbcf166e6f74ef224d4a63be4277d017bb62e minikube.k8s.io/name=minikube minikube.k8s.io/updated_at=2021_08_21T14_48_36_0700 --all --overwrite --kubeconfig=/var/lib/minikube/kubeconfig
I0821 14:48:37.086282 3693658 ops.go:34] apiserver oom_adj: -16
I0821 14:48:37.086313 3693658 kubeadm.go:985] duration metric: took 178.937976ms to wait for elevateKubeSystemPrivileges.
I0821 14:48:37.278793 3693658 kubeadm.go:392] StartCluster complete in 21.088866951s
I0821 14:48:37.278830 3693658 settings.go:142] acquiring lock: {Name:mkdae8f3141afc32430f0ccedd7ee5abbd89dc87 Clock:{} Delay:500ms Timeout:1m0s Cancel:
[Aug 5 08:26] ACPI: RSDP 00000000000f57e0 00014 (v00 BOCHS )
[ +0.000000] ACPI: RSDT 00000000bffe1b30 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ +0.000000] ACPI: FACP 00000000bffe0bfe 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ +0.000000] ACPI: DSDT 00000000bffe0040 00BBE (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
[ +0.000000] ACPI: FACS 00000000bffe0000 00040
[ +0.000000] ACPI: SSDT 00000000bffe0c72 00D46 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
[ +0.000000] ACPI: APIC 00000000bffe19b8 00090 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ +0.000000] ACPI: SRAT 00000000bffe1a48 000E8 (v01 BOCHS BXPCSRAT 00000001 BXPC 00000001)
[ +0.000000] Zone ranges:
[ +0.000000] DMA [mem 0x00001000-0x00ffffff]
[ +0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ +0.000000] Normal [mem 0x100000000-0x23fffffff]
[ +0.000000] Movable zone start for each node
[ +0.000000] Early memory node ranges
[ +0.000000] node 0: [mem 0x00001000-0x0009efff]
[ +0.000000] node 0: [mem 0x00100000-0xbffdffff]
[ +0.000000] node 0: [mem 0x100000000-0x23fffffff]
[ +0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2064233
[ +0.000000] Policy zone: Normal
[ +0.000000] ACPI: All ACPI Tables successfully acquired
[ +0.086531] ACPI: Executed 2 blocks of module-level executable AML code
[ +0.000714] ACPI: Enabled 16 GPEs in block 00 to 0F
[ +0.002332] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[ +0.106394] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[ +0.301595] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ +0.385110] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
[ +0.091306] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
[ +0.078787] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[Aug 5 08:29] AliSecGuard: loading out-of-tree module taints kernel.
[Aug 5 08:36] atkbd serio0: Unknown key pressed (translated set 2, code 0x0 on isa0060/serio0).
[ +0.000005] atkbd serio0: Use 'setkeycodes 00
MemoryPressure False Sat, 21 Aug 2021 07:00:18 +0000 Sat, 21 Aug 2021 06:48:30 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sat, 21 Aug 2021 07:00:18 +0000 Sat, 21 Aug 2021 06:48:30 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sat, 21 Aug 2021 07:00:18 +0000 Sat, 21 Aug 2021 06:48:30 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Sat, 21 Aug 2021 07:00:18 +0000 Sat, 21 Aug 2021 06:48:33 +0000 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.49.2 Hostname: minikube Capacity: cpu: 4 ephemeral-storage: 103078876Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 8008872Ki pods: 110 Allocatable: cpu: 4 ephemeral-storage: 103078876Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 8008872Ki pods: 110 System Info: Machine ID: 760e67beb8554645829f2357c8eb4ae7 System UUID: 1a44498b-0c8a-42d5-8b32-2fff4c89e1a7 Boot ID: b7219046-486f-42e7-a904-4b212a079609 Kernel Version: 3.10.0-957.5.1.el7.x86_64 OS Image: Ubuntu 20.04.2 LTS Operating System: linux Architecture: amd64 Container Runtime Version: docker://20.10.7 Kubelet Version: v1.21.2 Kube-Proxy Version: v1.21.2 PodCIDR: 10.244.0.0/24 PodCIDRs: 10.244.0.0/24 Non-terminated Pods: (8 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
ingress-nginx ingress-nginx-controller-59b45fb494-2tcfb 100m (2%) 0 (0%) 90Mi (1%) 0 (0%) 11m kube-system coredns-558bd4d5db-hh5hv 100m (2%) 0 (0%) 70Mi (0%) 170Mi (2%) 11m kube-system etcd-minikube 100m (2%) 0 (0%) 100Mi (1%) 0 (0%) 11m kube-system kube-apiserver-minikube 250m (6%) 0 (0%) 0 (0%) 0 (0%) 11m kube-system kube-controller-manager-minikube 200m (5%) 0 (0%) 0 (0%) 0 (0%) 11m kube-system kube-proxy-9jpc8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 11m kube-system kube-scheduler-minikube 100m (2%) 0 (0%) 0 (0%) 0 (0%) 11m kube-system storage-provisioner 0 (0%) 0 (0%) 0 (0%) 0 (0%) 11m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits
cpu 850m (21%) 0 (0%) memory 260Mi (3%) 170Mi (2%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: Type Reason Age From Message
Normal NodeHasSufficientMemory 11m (x5 over 11m) kubelet Node minikube status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 11m (x5 over 11m) kubelet Node minikube status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 11m (x4 over 11m) kubelet Node minikube status is now: NodeHasSufficientPID Normal Starting 11m kubelet Starting kubelet. Normal NodeHasSufficientMemory 11m kubelet Node minikube status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 11m kubelet Node minikube status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 11m kubelet Node minikube status is now: NodeHasSufficientPID Normal NodeAllocatableEnforced 11m kubelet Updated Node Allocatable limit across pods Normal Starting 11m kube-proxy Starting kube-proxy. Aug 21 07:00:23 minikube kubelet[2346]: E0821 07:00:23.264860 2346 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"controller\" with ImagePullBackOff: \"Back-off pulling image \\"k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a\\"\"" pod="ingress-nginx/ingress-nginx-controller-59b45fb494-2tcfb" podUID=b658f80d-553d-4ceb-b612-02052ef45fad Aug 21 07:00:38 minikube kubelet[2346]: E0821 07:00:38.264157 2346 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"controller\" with ImagePullBackOff: \"Back-off pulling image \\"k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a\\"\"" pod="ingress-nginx/ingress-nginx-controller-59b45fb494-2tcfb" podUID=b658f80d-553d-4ceb-b612-02052ef45fad Aug 21 07:00:49 minikube kubelet[2346]: E0821 07:00:49.264324 2346 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"controller\" with ImagePullBackOff: \"Back-off pulling image \\"k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a\\"\"" pod="ingress-nginx/ingress-nginx-controller-59b45fb494-2tcfb" podUID=b658f80d-553d-4ceb-b612-02052ef45fad Aug 21 07:01:00 minikube kubelet[2346]: E0821 07:01:00.264173 2346 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"controller\" with ImagePullBackOff: \"Back-off pulling image \\"k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a\\"\"" pod="ingress-nginx/ingress-nginx-controller-59b45fb494-2tcfb" podUID=b658f80d-553d-4ceb-b612-02052ef45fad Aug 21 07:01:14 minikube kubelet[2346]: E0821 07:01:14.263454 2346 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"controller\" with ImagePullBackOff: \"Back-off pulling image \\"k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a\\"\"" pod="ingress-nginx/ingress-nginx-controller-59b45fb494-2tcfb" podUID=b658f80d-553d-4ceb-b612-02052ef45fad Aug 21 07:01:25 minikube kubelet[2346]: E0821 07:01:25.263352 2346 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"controller\" with ImagePullBackOff: \"Back-off pulling image \\"k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a\\"\"" pod="ingress-nginx/ingress-nginx-controller-59b45fb494-2tcfb" podUID=b658f80d-553d-4ceb-b612-02052ef45fad Aug 21 07:01:36 minikube kubelet[2346]: E0821 07:01:36.264560 2346 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"controller\" with ImagePullBackOff: \"Back-off pulling image \\"k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a\\"\"" pod="ingress-nginx/ingress-nginx-controller-59b45fb494-2tcfb" podUID=b658f80d-553d-4ceb-b612-02052ef45fad Aug 21 07:01:50 minikube kubelet[2346]: E0821 07:01:50.264303 2346 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"controller\" with ImagePullBackOff: \"Back-off pulling image \\"k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a\\"\"" pod="ingress-nginx/ingress-nginx-controller-59b45fb494-2tcfb" podUID=b658f80d-553d-4ceb-b612-02052ef45fad
jackhenhao
commented
3 years ago 重现问题所需的命令：minikube addons enable ingress
失败的命令的完整输出：
minikube logs命令的输出：使用的操作系统版本： ubuntu20.04
重现问题所需的命令：minikube addons enable ingress
失败的命令的完整输出：
▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
🔎 Verifying ingress addon...
❌ Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
╭─────────────────────────────────────────────────────────────────────────────╮ │ │ │ 😿 If the above advice does not help, please let us know: │ │ 👉 https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please attach the following file to the GitHub issue: │ │ - /tmp/minikube_addons_bf1031aaca0ed6f7b83a08d9bf0cd116bd9110a7_0.log │ │ │ ╰─────────────────────────────────────────────────────────────────────────────╯
minikube logs命令的输出：使用的操作系统版本： ubuntu20.04