search

rancher / rio

Application Deployment Engine for Kubernetes
https://rio.io
Apache License 2.0
2.27k stars 228 forks source link

Rio stucked on reinstall #1025

Closed citananda closed 4 years ago

citananda commented 4 years ago

Describe the bug After rio uninstall, rio install is stucked on:

Deploying Rio control plane....
| Waiting for system components: [rio-system/cert-manager rio-system/gateway rio-system/gateway-proxy ...]

To Reproduce rio uninstall rio install The result is the same with rio 0.7.0 and 0.7.1-rc1

Expected behavior rio installed successful

Kubernetes version & type (GKE, on-prem): kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Type: Rio version: rio info

FATA[0000] no system information is generated

but the bug is the same with rio 0.7.0 and 0.7.1-rc1 Additional context rio system logs output:

rio-controller | time="2020-04-09T14:46:04Z" level=info msg="Starting rio-controller, version: v0.7.1-rc1, git commit: 121e5842"
rio-controller | time="2020-04-09T14:46:04Z" level=info msg="Updating CRD services.rio.cattle.io"
rio-controller | time="2020-04-09T14:46:04Z" level=info msg="Updating CRD stacks.rio.cattle.io"
rio-controller | I0409 14:46:04.928641       1 leaderelection.go:241] attempting to acquire leader lease  rio-system/rio...
rio-controller | time="2020-04-09T14:46:04Z" level=info msg="listening at :443"
rio-controller | I0409 14:46:04.969777       1 leaderelection.go:251] successfully acquired lease rio-system/rio
rio-controller | time="2020-04-09T14:46:07Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting apps/v1, Kind=Deployment controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting rio.cattle.io/v1, Kind=Router controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting gloo.solo.io/v1, Kind=Settings controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting apps/v1, Kind=DaemonSet controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting apps/v1, Kind=StatefulSet controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting /v1, Kind=Service controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting /v1, Kind=Endpoints controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting /v1, Kind=Secret controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting cert-manager.io/v1alpha2, Kind=Certificate controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting extensions/v1beta1, Kind=Ingress controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting admin.rio.cattle.io/v1, Kind=ClusterDomain controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting admin.rio.cattle.io/v1, Kind=PublicDomain controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting rio.cattle.io/v1, Kind=Service controller"
rio-controller | time="2020-04-09T14:46:15Z" level=info msg="Starting rio.cattle.io/v1, Kind=ExternalService controller"
citananda commented 4 years ago

image

citananda commented 4 years ago

And here is all the process logs rio -n rio-system logs -a

+ rio-system rio-controller-597fb9d959-47n7x › rio-controller
Error opening stream to rio-system/rio-controller-597fb9d959-47n7x: rio-controller
: container "rio-controller" in pod "rio-controller-597fb9d959-47n7x" is waiting to start: ContainerCreating
- rio-system rio-controller-597fb9d959-47n7x
+ rio-system rio-controller-597fb9d959-47n7x › rio-controller
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:01Z" level=info msg="Starting rio-controller, version: v0.7.0, git commit: 4afd4901"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:01Z" level=info msg="Creating CRD externalservices.rio.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:01Z" level=info msg="Creating CRD routers.rio.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:01Z" level=info msg="Creating CRD services.rio.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:01Z" level=info msg="Creating CRD stacks.rio.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:01Z" level=info msg="Creating CRD clusterdomains.admin.rio.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:01Z" level=info msg="Creating CRD publicdomains.admin.rio.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:02Z" level=info msg="Creating CRD rioinfos.admin.rio.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:02Z" level=info msg="Creating CRD systemstacks.admin.rio.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:02Z" level=info msg="Creating CRD gitcommits.gitwatcher.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:02Z" level=info msg="Creating CRD gitwatchers.gitwatcher.cattle.io"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:02Z" level=info msg="Waiting for CRD gitwatchers.gitwatcher.cattle.io to become available"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:03Z" level=info msg="Done waiting for CRD gitwatchers.gitwatcher.cattle.io to become available"
rio-controller-597fb9d959-47n7x rio-controller time="2020-04-09T17:07:05Z" level=fatal msg="failed to read TLS secrets: open /var/run/rio/ssl/tls.key: no such file or directory"
+ rio-system linkerd-install-rntgd-rk74b › linkerd-install
Error opening stream to rio-system/linkerd-install-rntgd-rk74b: linkerd-install
: container "linkerd-install" in pod "linkerd-install-rntgd-rk74b" is waiting to start: ContainerCreating
- rio-system linkerd-install-rntgd-rk74b
+ rio-system linkerd-install-rntgd-rk74b › linkerd-install
Error opening stream to rio-system/linkerd-install-rntgd-rk74b: linkerd-install
: container "linkerd-install" in pod "linkerd-install-rntgd-rk74b" is waiting to start: ContainerCreating
- rio-system linkerd-install-rntgd-rk74b
+ rio-system linkerd-install-rntgd-rk74b › linkerd-install
linkerd-install-rntgd-rk74b linkerd-install + [[ -n '' ]]
linkerd-install-rntgd-rk74b linkerd-install + [[ -n '' ]]
linkerd-install-rntgd-rk74b linkerd-install + linkerd install
linkerd-install-rntgd-rk74b linkerd-install + kubectl apply -f -
linkerd-install-rntgd-rk74b linkerd-install namespace/linkerd created
linkerd-install-rntgd-rk74b linkerd-install clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-identity created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-identity created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-identity created
linkerd-install-rntgd-rk74b linkerd-install clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-controller created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-controller created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-controller created
linkerd-install-rntgd-rk74b linkerd-install clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-destination created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-destination created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-destination created
linkerd-install-rntgd-rk74b linkerd-install role.rbac.authorization.k8s.io/linkerd-heartbeat created
linkerd-install-rntgd-rk74b linkerd-install rolebinding.rbac.authorization.k8s.io/linkerd-heartbeat created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-heartbeat created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-web-admin created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-web created
linkerd-install-rntgd-rk74b linkerd-install customresourcedefinition.apiextensions.k8s.io/serviceprofiles.linkerd.io created
linkerd-install-rntgd-rk74b linkerd-install Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
linkerd-install-rntgd-rk74b linkerd-install customresourcedefinition.apiextensions.k8s.io/trafficsplits.split.smi-spec.io configured
linkerd-install-rntgd-rk74b linkerd-install clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-prometheus created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-prometheus created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-prometheus created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-grafana created
linkerd-install-rntgd-rk74b linkerd-install clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-proxy-injector created
linkerd-install-rntgd-rk74b linkerd-install secret/linkerd-proxy-injector-tls created
linkerd-install-rntgd-rk74b linkerd-install mutatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-proxy-injector-webhook-config created
linkerd-install-rntgd-rk74b linkerd-install clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-sp-validator created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-sp-validator created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-sp-validator created
linkerd-install-rntgd-rk74b linkerd-install secret/linkerd-sp-validator-tls created
linkerd-install-rntgd-rk74b linkerd-install validatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-sp-validator-webhook-config created
linkerd-install-rntgd-rk74b linkerd-install clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-tap created
linkerd-install-rntgd-rk74b linkerd-install clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-tap-admin created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-tap created
linkerd-install-rntgd-rk74b linkerd-install clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-tap-auth-delegator created
linkerd-install-rntgd-rk74b linkerd-install serviceaccount/linkerd-tap created
linkerd-install-rntgd-rk74b linkerd-install rolebinding.rbac.authorization.k8s.io/linkerd-linkerd-tap-auth-reader created
linkerd-install-rntgd-rk74b linkerd-install secret/linkerd-tap-tls created
linkerd-install-rntgd-rk74b linkerd-install apiservice.apiregistration.k8s.io/v1alpha1.tap.linkerd.io created
linkerd-install-rntgd-rk74b linkerd-install podsecuritypolicy.policy/linkerd-linkerd-control-plane created
linkerd-install-rntgd-rk74b linkerd-install role.rbac.authorization.k8s.io/linkerd-psp created
linkerd-install-rntgd-rk74b linkerd-install rolebinding.rbac.authorization.k8s.io/linkerd-psp created
linkerd-install-rntgd-rk74b linkerd-install configmap/linkerd-config created
linkerd-install-rntgd-rk74b linkerd-install secret/linkerd-identity-issuer created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-identity created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-identity created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-controller-api created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-destination created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-controller created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-dst created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-destination created
linkerd-install-rntgd-rk74b linkerd-install cronjob.batch/linkerd-heartbeat created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-web created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-web created
linkerd-install-rntgd-rk74b linkerd-install configmap/linkerd-prometheus-config created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-prometheus created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-prometheus created
linkerd-install-rntgd-rk74b linkerd-install configmap/linkerd-grafana-config created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-grafana created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-grafana created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-proxy-injector created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-proxy-injector created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-sp-validator created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-sp-validator created
linkerd-install-rntgd-rk74b linkerd-install service/linkerd-tap created
linkerd-install-rntgd-rk74b linkerd-install deployment.apps/linkerd-tap created
linkerd-install-rntgd-rk74b linkerd-install + linkerd check
linkerd-install-rntgd-rk74b linkerd-install kubernetes-api
linkerd-install-rntgd-rk74b linkerd-install --------------
linkerd-install-rntgd-rk74b linkerd-install √ can initialize the client
linkerd-install-rntgd-rk74b linkerd-install √ can query the Kubernetes API
linkerd-install-rntgd-rk74b linkerd-install 
linkerd-install-rntgd-rk74b linkerd-install kubernetes-version
linkerd-install-rntgd-rk74b linkerd-install ------------------
linkerd-install-rntgd-rk74b linkerd-install √ is running the minimum Kubernetes API version
linkerd-install-rntgd-rk74b linkerd-install √ is running the minimum kubectl version
linkerd-install-rntgd-rk74b linkerd-install 
linkerd-install-rntgd-rk74b linkerd-install linkerd-config
linkerd-install-rntgd-rk74b linkerd-install --------------
linkerd-install-rntgd-rk74b linkerd-install √ control plane Namespace exists
linkerd-install-rntgd-rk74b linkerd-install √ control plane ClusterRoles exist
linkerd-install-rntgd-rk74b linkerd-install √ control plane ClusterRoleBindings exist
linkerd-install-rntgd-rk74b linkerd-install √ control plane ServiceAccounts exist
linkerd-install-rntgd-rk74b linkerd-install √ control plane CustomResourceDefinitions exist
linkerd-install-rntgd-rk74b linkerd-install √ control plane MutatingWebhookConfigurations exist
linkerd-install-rntgd-rk74b linkerd-install √ control plane ValidatingWebhookConfigurations exist
linkerd-install-rntgd-rk74b linkerd-install √ control plane PodSecurityPolicies exist
linkerd-install-rntgd-rk74b linkerd-install 
linkerd-install-rntgd-rk74b linkerd-install linkerd-existence
linkerd-install-rntgd-rk74b linkerd-install -----------------
linkerd-install-rntgd-rk74b linkerd-install √ 'linkerd-config' config map exists
linkerd-install-rntgd-rk74b linkerd-install √ heartbeat ServiceAccount exist
linkerd-install-rntgd-rk74b linkerd-install √ control plane replica sets are ready
linkerd-install-rntgd-rk74b linkerd-install √ no unschedulable pods
linkerd-install-rntgd-rk74b linkerd-install √ controller pod is running
linkerd-install-rntgd-rk74b linkerd-install √ can initialize the client
linkerd-install-rntgd-rk74b linkerd-install √ can query the control plane API
linkerd-install-rntgd-rk74b linkerd-install 
linkerd-install-rntgd-rk74b linkerd-install linkerd-api
linkerd-install-rntgd-rk74b linkerd-install -----------
linkerd-install-rntgd-rk74b linkerd-install √ control plane pods are ready
linkerd-install-rntgd-rk74b linkerd-install √ control plane self-check
linkerd-install-rntgd-rk74b linkerd-install √ [kubernetes] control plane can talk to Kubernetes
linkerd-install-rntgd-rk74b linkerd-install √ [prometheus] control plane can talk to Prometheus
linkerd-install-rntgd-rk74b linkerd-install √ no invalid service profiles
linkerd-install-rntgd-rk74b linkerd-install 
linkerd-install-rntgd-rk74b linkerd-install linkerd-version
linkerd-install-rntgd-rk74b linkerd-install ---------------
linkerd-install-rntgd-rk74b linkerd-install √ can determine the latest version
linkerd-install-rntgd-rk74b linkerd-install ‼ cli is up-to-date
linkerd-install-rntgd-rk74b linkerd-install     is running version 2.6.1 but the latest stable version is 2.7.0
linkerd-install-rntgd-rk74b linkerd-install     see https://linkerd.io/checks/#l5d-version-cli for hints
linkerd-install-rntgd-rk74b linkerd-install 
linkerd-install-rntgd-rk74b linkerd-install control-plane-version
linkerd-install-rntgd-rk74b linkerd-install ---------------------
linkerd-install-rntgd-rk74b linkerd-install ‼ control plane is up-to-date
linkerd-install-rntgd-rk74b linkerd-install     is running version 2.6.1 but the latest stable version is 2.7.0
linkerd-install-rntgd-rk74b linkerd-install     see https://linkerd.io/checks/#l5d-version-control for hints
linkerd-install-rntgd-rk74b linkerd-install √ control plane and cli versions match
linkerd-install-rntgd-rk74b linkerd-install 
linkerd-install-rntgd-rk74b linkerd-install Status check results are √
linkerd-install-rntgd-rk74b linkerd-install + [[ 0 -ne 0 ]]
citananda commented 4 years ago

The solution was to move namespaces linkerd and rio-system to the project system and then to redeploy the stucked workloads