pull-kubernetes-dns-test broken at head

[jingyuanliang]

Found in: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/dns/603/pull-kubernetes-dns-test/1715355423860592640

Confirmed failure without code change at all: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/dns/604/pull-kubernetes-dns-test/1716792265403273216 (1716792265403273216 #604 (eaaed71) by jingyuanliang 08:22:35 26m57s FAILURE)

Manipulated test code: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/dns/604/pull-kubernetes-dns-test/1716834170107334656 (1716834170107334656 #604 (68b664b) by jingyuanliang 11:09:05 26m59s SUCCESS)

And extracted the failure:

2023/10/24 15:35:43 LOG: docker | Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
2023/10/24 15:35:43 LOG: docker | Flag --config has been deprecated, Use --pod-manifest-path instead. Will be removed in a future version.
2023/10/24 15:35:43 LOG: docker | I1024 15:35:22.119199  199402 feature_gate.go:181] feature gates: map[]
2023/10/24 15:35:43 LOG: docker | W1024 15:35:22.119532  199402 server.go:605] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Using default client config instead.
2023/10/24 15:35:43 LOG: docker | I1024 15:35:22.665666  199402 server.go:536] Using self-signed cert (/var/run/kubernetes/kubelet.crt, /var/run/kubernetes/kubelet.key)
2023/10/24 15:35:43 LOG: docker | I1024 15:35:22.665796  199402 server.go:124] Running kubelet in containerized mode (experimental)
2023/10/24 15:35:43 LOG: docker | I1024 15:35:22.665986  199402 docker.go:356] Connecting to docker on unix:///var/run/docker.sock
2023/10/24 15:35:43 LOG: docker | I1024 15:35:22.666057  199402 docker.go:376] Start docker client with request timeout=2m0s
2023/10/24 15:35:43 LOG: docker | I1024 15:35:22.680891  199402 iptables.go:176] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory
2023/10/24 15:35:43 LOG: docker | I1024 15:35:22.683548  199402 iptables.go:176] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory
2023/10/24 15:35:43 LOG: docker | Error: failed to run Kubelet: mountpoint for cpu not found
2023/10/24 15:35:43 LOG: docker | failed to run Kubelet: mountpoint for cpu not found

cc @wewark

[jingyuanliang]

Interestingly, with kubelet failed, the tests can still pass, and the tests failed only when we try to killed kubelet (and found it already terminated)

LOG: docker [-H unix:///var/run/docker.sock kill 3f4260312b43f505b7d3c9c3dbdcc3e6367a0e67f276dbef07aea25f6cc438fb]
2023/10/24 15:35:43 LOG: docker | Error response from daemon: Cannot kill container: 3f4260312b43f505b7d3c9c3dbdcc3e6367a0e67f276dbef07aea25f6cc438fb: Container 3f4260312b43f505b7d3c9c3dbdcc3e6367a0e67f276dbef07aea25f6cc438fb is not running

[jingyuanliang]

Ref: startup command:

2023/10/24 15:35:21 LOG: docker | 
LOG: docker run [-H unix:///var/run/docker.sock run -d --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/dev:/dev --volume=/home/prow/go/src/k8s.io/dns:/src:ro --volume=/tmp/k8s-dns:/data:rw --volume=/home/prow/go/src/k8s.io/dns/test/e2e/cluster/manifests:/etc/kubernetes/manifests-e2e:ro --volume=/var/lib/docker:/var/lib/docker:rw --volume=/var/run:/var/run:rw --volume=/var/lib/kubelet:/var/lib/kubelet:shared --net=host --pid=host --privileged=true registry.k8s.io/hyperkube:v1.5.1 /hyperkube kubelet --v=4 --containerized --hostname-override=0.0.0.0 --address=0.0.0.0 --cluster_dns=10.0.0.10 --cluster_domain=cluster.local --api-servers=http://localhost:8080 --config=/etc/kubernetes/manifests-e2e]