search

openshift / installer

Install an OpenShift 4.x cluster
https://try.openshift.com
Apache License 2.0
1.44k stars 1.39k forks source link

OCP 4.7 UPI Installation Error. Pods in bootstrap exit immediatly after creating #5311

Closed SunnyGu74 closed 3 years ago

SunnyGu74 commented 3 years ago

Version

$ openshift-install version
[root@bastion ocpinstall]# ./openshift-install version
./openshift-install 4.7.32
built from commit 94125b21304a574ae5bc98039f8eb7f518293b83
release image quay.io/openshift-release-dev/ocp-release@sha256:96f00e5b0cde92488c42cd9395ff5a755c77f4273eb10aa83bc2707a8badf93f

Platform:

baremetal UPI

What happened?

keep seeing following event from bootkube. Oct 19 03:15:59 bootstrap.ocp47u.sunnylab.com bootkube.sh[11870]: Skipped "secret-kube-apiserver-to-kubelet-signer.yaml" secrets.v1./kube-apiserver-to>Oct 19 03:15:59 bootstrap.ocp47u.sunnylab.com bootkube.sh[11870]: Skipped "secret-loadbalancer-serving-signer.yaml" secrets.v1./loadbalancer-serving-s>Oct 19 03:16:00 bootstrap.ocp47u.sunnylab.com bootkube.sh[11870]: Skipped "secret-localhost-serving-signer.yaml" secrets.v1./localhost-serving-signer > Oct 19 03:16:00 bootstrap.ocp47u.sunnylab.com bootkube.sh[11870]: Skipped "secret-service-network-serving-signer.yaml" secrets.v1./service-network-ser> Oct 19 03:35:27 bootstrap.ocp47u.sunnylab.com bootkube.sh[11870]: Error: error while checking pod status: timed out waiting for the condition Oct 19 03:35:27 bootstrap.ocp47u.sunnylab.com bootkube.sh[11870]: Tearing down temporary bootstrap control plane... Oct 19 03:35:27 bootstrap.ocp47u.sunnylab.com bootkube.sh[11870]: Error: error while checking pod status: timed out waiting for the condition Oct 19 03:35:27 bootstrap.ocp47u.sunnylab.com systemd[1]: bootkube.service: Main process exited, code=exited, status=1/FAILURE Oct 19 03:35:27 bootstrap.ocp47u.sunnylab.com systemd[1]: bootkube.service: Failed with result 'exit-code'. Oct 19 03:35:32 bootstrap.ocp47u.sunnylab.com systemd[1]: bootkube.service: Service RestartSec=5s expired, scheduling restart. Oct 19 03:35:32 bootstrap.ocp47u.sunnylab.com systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 2. Oct 19 03:35:32 bootstrap.ocp47u.sunnylab.com systemd[1]: Stopped Bootstrap a Kubernetes cluster. Oct 19 03:35:32 bootstrap.ocp47u.sunnylab.com systemd[1]: Started Bootstrap a Kubernetes cluster. Oct 19 03:35:47 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: https://localhost:2379 is healthy: successfully committed proposal: took = 14.749635> Oct 19 03:35:47 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Starting cluster-bootstrap... Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Starting temporary bootstrap control plane... Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_00_cluster-version-operator_00_namespace.yaml" namespaces.v1./openshif> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:48.630337 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_00_cluster-version-operator_01_clusteroperator.crd.yaml" customresourc> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:48.633491 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:48.650416 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_00_cluster-version-operator_01_clusterversion.crd.yaml" customresource> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:48.653739 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_00_cluster-version-operator_02_roles.yaml" clusterrolebindings.v1.rbac> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_00_cluster-version-operator_03_deployment.yaml" deployments.v1.apps/cl> Oct 19 03:35:48 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_03_authorization-openshift_01_rolebindingrestriction.crd.yaml" customr>Oct 19 03:35:49 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:49.218601 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:49 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_03_config-operator_01_operatorhub.crd.yaml" customresourcedefinitions.> Oct 19 03:35:49 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:49.413766 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:49 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_03_config-operator_01_proxy.crd.yaml" customresourcedefinitions.v1.api> Oct 19 03:35:50 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:50.019366 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:50 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_03_quota-openshift_01_clusterresourcequota.crd.yaml" customresourcedef> Oct 19 03:35:50 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:50.214058 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:50 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_03_security-openshift_01_scc.crd.yaml" customresourcedefinitions.v1.ap> Oct 19 03:35:50 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_03_securityinternal-openshift_02_rangeallocation.crd.yaml" customresou> Oct 19 03:35:51 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_10_config-operator_01_apiserver.crd.yaml" customresourcedefinitions.v1> Oct 19 03:35:51 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_10_config-operator_01_authentication.crd.yaml" customresourcedefinitio> Oct 19 03:35:52 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:52.034922 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes> Oct 19 03:35:52 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: Skipped "0000_10_config-operator_01_build.crd.yaml" customresourcedefinitions.v1beta> Oct 19 03:35:52 bootstrap.ocp47u.sunnylab.com bootkube.sh[19215]: W1019 03:35:52.220119 1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomRes>

Only one container is running, others are exited after creating. [core@bootstrap ~]$ sudo podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d74317077f38 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4aaab06c42efed4fa94bc833b4269024a5167cd1267217c293343f8e25c97a47 start --tear-down... 17 minutes ago Up 17 minutes ago relaxed_wing 1b5ea9e97deb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e44b05fbfbd5917b24e20c42b11aa414eed575516fc181e91f7dd99f3f58c18e render --dest-dir... 58 minutes ago Exited (0) 58 minutes ago fervent_rhodes ea2f16d1adc1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8108896fec6efaa091b941139a963913349f86034c6147945fe615fd9a31b33 bootstrap --root-... 59 minutes ago Exited (0) 59 minutes ago angry_gagarin 51ca85df0ea9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5fef4fe158b899a720e5b724409c49e58d46c74855b164a73194d02dbe4a9438 render --prefix=c... About an hour ago Exited (0) About an hour ago quirky_kare ce7a1db85e94 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a09ad0a483f187183d3a6d862edd96b5f12dfc977b517266e52e5c19e66e2eeb /usr/bin/cluster-... About an hour ago Exited (0) About an hour ago amazing_napier 84a4ab71283d quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39e3db92ba2944b1770893fb81de3415252b5aee217f7762f83ed724ec13c058 /usr/bin/cluster-... About an hour ago Exited (0) About an hour ago stoic_elion 2870d5d7724d quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a60c78b9f9bd1dbadf7e7c5f0e1aa5ab3575a1da8fc30f3903f6bfc7589b344b /usr/bin/cluster-... About an hour ago Exited (0) About an hour ago charming_maxwell d59ee47ad71b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0a1139d422dd165f126151f87df7ac4a518974fbbf6f70f18dfd12b1cd14b58 /usr/bin/cluster-... About an hour ago Exited (0) About an hour ago vigorous_hertz 0b0a071750b5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:68fd3837d821efd8ef5ceb852f8fe8a28a29fb5bedc66f6a26dd58adbb43cfe0 /usr/bin/cluster-... About an hour ago Exited (0) About an hour ago great_mendeleev 67f48bb43494 quay.io/openshift-release-dev/ocp-release@sha256:96f00e5b0cde92488c42cd9395ff5a755c77f4273eb10aa83bc2707a8badf93f render --output-d... About an hour ago Exited (0) About an hour ago determined_galois

And see this error Oct 19 03:30:27 bootstrap.ocp47u.sunnylab.com hyperkube[2351]: E1019 03:30:27.177924 2351 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"

What you expected to happen?

Complete bootstrap installation

How to reproduce it (as minimally and precisely as possible)?

Follow RH guide to install it. here is install-config.yaml file apiVersion: v1 baseDomain: sunnylab.com metadata: name: ocp47u compute:

$ your-commands-here

Anything else we need to know?

Enter text here.

References

SunnyGu74 commented 3 years ago

log-bundle-20211019115130.tar.gz

SunnyGu74 commented 3 years ago

BTW, I use VMs in vsphere environment install the OCP 4.7.

SunnyGu74 commented 3 years ago

anyone can help me?

SunnyGu74 commented 3 years ago

oh, I just notice I can install master node on current status. it seems not issue.

SunnyGu74 commented 3 years ago

This is not issue.