failed to initialize the cluster

okovalchuk777 commented 3 years ago

Describe the bug I try to have UPI (bare metal) installation of stable release OKD (vSphere). If I try to deploy a cluster (releases OKD 4.6.0-0.okd-2021-01-17-185703, OKD 4.6.0-0.okd-2021-01-23-132511), I get an error "FATAL failed to initialize the cluster: Multiple errors are preventing progress"

Version Client Version: 4.6.0-0.okd-2021-01-23-132511 Server Version: 4.6.0-0.okd-2021-01-23-132511 Kubernetes Version: v1.19.2-1023+4cad5caeae67c3-dirty

oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication False True True 19h cloud-credential 4.6.0-0.okd-2021-01-23-132511 True False False 20h cluster-autoscaler 4.6.0-0.okd-2021-01-23-132511 True False False 19h config-operator 4.6.0-0.okd-2021-01-23-132511 True False False 19h console 4.6.0-0.okd-2021-01-23-132511 False False True 4h27m csi-snapshot-controller 4.6.0-0.okd-2021-01-23-132511 True False False 19h dns 4.6.0-0.okd-2021-01-23-132511 True False False 19h etcd 4.6.0-0.okd-2021-01-23-132511 True False False 19h image-registry 4.6.0-0.okd-2021-01-23-132511 True False True 19h ingress 4.6.0-0.okd-2021-01-23-132511 True False False 19h insights 4.6.0-0.okd-2021-01-23-132511 True False False 19h kube-apiserver 4.6.0-0.okd-2021-01-23-132511 True False False 19h kube-controller-manager 4.6.0-0.okd-2021-01-23-132511 True False False 19h kube-scheduler 4.6.0-0.okd-2021-01-23-132511 True False False 19h kube-storage-version-migrator 4.6.0-0.okd-2021-01-23-132511 True False False 19h machine-api 4.6.0-0.okd-2021-01-23-132511 True False False 19h machine-approver 4.6.0-0.okd-2021-01-23-132511 True False False 19h machine-config 4.6.0-0.okd-2021-01-23-132511 True False False 19h marketplace 4.6.0-0.okd-2021-01-23-132511 True False False 19h monitoring False False True 19h network 4.6.0-0.okd-2021-01-23-132511 True False False 19h node-tuning 4.6.0-0.okd-2021-01-23-132511 True False False 19h openshift-apiserver 4.6.0-0.okd-2021-01-23-132511 False False False 19h openshift-controller-manager 4.6.0-0.okd-2021-01-23-132511 True True False 19h openshift-samples False False False 18h operator-lifecycle-manager 4.6.0-0.okd-2021-01-23-132511 True False False 19h operator-lifecycle-manager-catalog 4.6.0-0.okd-2021-01-23-132511 True False False 19h operator-lifecycle-manager-packageserver 4.6.0-0.okd-2021-01-23-132511 False True False 86m service-ca 4.6.0-0.okd-2021-01-23-132511 True False False 19h storage 4.6.0-0.okd-2021-01-23-132511 True False False 19h

Log bundle

openshift-install --dir=install_dir4/ wait-for bootstrap-complete --log-level=debug DEBUG OpenShift Installer 4.6.0-0.okd-2021-01-23-132511 DEBUG Built from commit 6f79a4c4279ce2b56a522d98ae1ea6fef2e02e9b INFO Waiting up to 20m0s for the Kubernetes API at https://api.okd4-cl-02.openshift.pravex.ua:6443... INFO API v1.19.2-1023+4cad5caeae67c3-dirty up INFO Waiting up to 30m0s for bootstrapping to complete... DEBUG Bootstrap status: complete INFO It is now safe to remove the bootstrap resources INFO Time elapsed: 0s

openshift-install --dir=install_dir4/ wait-for install-complete --log-level debug DEBUG OpenShift Installer 4.6.0-0.okd-2021-01-23-132511 DEBUG Built from commit 6f79a4c4279ce2b56a522d98ae1ea6fef2e02e9b DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Using Install Config loaded from state file INFO Waiting up to 40m0s for the cluster at https://api.okd4-cl-02.xxxx:6443 to initialize... DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:

Could not update oauthclient "console" (385 of 627)
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (533 of 627)
Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (549 of 627)
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (340 of 627)
Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 627)
Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (277 of 627)
Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (240 of 627) DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-01-23-132511: 81% complete DEBUG Still waiting for the cluster to initialize: Could not update oauthclient "console" (385 of 627) DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
Could not update oauthclient "console" (385 of 627)
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (533 of 627)
Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (549 of 627)
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (340 of 627)
Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 627)
Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (277 of 627)
Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (240 of 627) DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-01-23-132511: 83% complete DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (533 of 627)
Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (549 of 627)
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (340 of 627)
Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 627)
Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (277 of 627)
Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (240 of 627)
Could not update route "openshift-console/downloads" (418 of 627) DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
Could not update oauthclient "console" (385 of 627)
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (533 of 627)
Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (549 of 627)
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (340 of 627)
Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 627)
Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (277 of 627)
Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (240 of 627) DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
Could not update oauthclient "console" (385 of 627)
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (533 of 627): the server is reporting an internal error
Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (549 of 627)
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (340 of 627): the server is reporting an internal error
Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 627)
Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (277 of 627): the server is reporting an internal error
Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (240 of 627) DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-01-23-132511: 83% complete DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (533 of 627)
Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (549 of 627)
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (340 of 627): the server is reporting an internal error
Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 627): the server is reporting an internal error
Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (277 of 627)
Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (240 of 627)
Could not update route "openshift-console/downloads" (418 of 627) ERROR Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthRouteCheckEndpointAccessibleController_SyncError::OAuthServerDeployment_GetFailed::OAuthServiceCheckEndpointAccessibleController_SyncError::OAuthServiceEndpointsCheckEndpointAccessibleController_SyncError::OAuthVersionDeployment_GetFailed: OAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready OAuthServiceCheckEndpointAccessibleControllerDegraded: Get "https://172.30.213.134:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server OAuthRouteCheckEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.okd4-cl-02.xxxx/healthz": EOF OAuthServerDeploymentDegraded: Unable to get "openshift-browser-client" bootstrapped OAuth client: the server is currently unable to handle the request (post oauthclients.oauth.openshift.io) OAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps "oauth-openshift" not found INFO Cluster operator authentication Available is False with APIServices_Error::OAuthRouteCheckEndpointAccessibleController_EndpointUnavailable::OAuthServiceCheckEndpointAccessibleController_EndpointUnavailable::OAuthServiceEndpointsCheckEndpointAccessibleController_EndpointUnavailable::OAuthVersionDeployment_MissingDeployment: OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints APIServicesAvailable: "oauth.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) APIServicesAvailable: "user.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) OAuthServiceCheckEndpointAccessibleControllerAvailable: Get "https://172.30.213.134:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) OAuthRouteCheckEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.okd4-cl-02.xxxx/healthz": EOF INFO Cluster operator insights Disabled is False with AsExpected: ERROR Cluster operator monitoring Degraded is True with UpdatingThanosQuerierFailed: Failed to rollout the stack. Error: running task Updating Thanos Querier failed: reconciling Thanos Querier Deployment failed: creating Deployment object failed: Internal error occurred: admission plugin "image.openshift.io/ImagePolicy" failed to complete mutation in 13s INFO Cluster operator monitoring Available is False with : INFO Cluster operator openshift-apiserver Available is False with APIServices_Error: APIServicesAvailable: "authorization.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) APIServicesAvailable: "build.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) APIServicesAvailable: "project.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) APIServicesAvailable: "quota.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) APIServicesAvailable: "security.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) APIServicesAvailable: "template.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) INFO Cluster operator openshift-samples Available is False with : FATAL failed to initialize the cluster: Multiple errors are preventing progress:
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (533 of 627)
Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (549 of 627)
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (340 of 627): the server is reporting an internal error
Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 627): the server is reporting an internal error
Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (277 of 627)
Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (240 of 627)
Could not update route "openshift-console/downloads" (418 of 627)

Please tell me what I'm doing wrong?

vrutkovs commented 3 years ago

Please attach log bundle

okovalchuk777 commented 3 years ago

Please, download link https://drive.google.com/file/d/1TCX9uNwOg9mpdFXuMqBoDABbZtX3r3RT/view?usp=sharing

vrutkovs commented 3 years ago

networkType: OpenShiftSDN

Dupe of #430

okovalchuk777 commented 3 years ago

Thanks!

ср, 27 янв. 2021 г., 10:33 Vadim Rutkovsky notifications@github.com:

Closed #485 https://github.com/openshift/okd/issues/485.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/openshift/okd/issues/485#event-4254585377, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARTCGMEG4XNCCW46W3TKTALS37FVTANCNFSM4WTZSXEA .

okd-project / okd

failed to initialize the cluster #485