search

banzaicloud / istio-operator

An operator that manages Istio deployments on Kubernetes
Apache License 2.0
535 stars 94 forks source link

Failed to reconcile resource could not reconcile istio #873

Open stephanie-tavares opened 2 years ago

stephanie-tavares commented 2 years ago

Hello guys,

I have a problem with istio using ingressgateway. I received this error log:

"timed out waiting for the condition\ncould not execute AfterUpdate func\ngithub.com/banzaicloud/istio-operator/pkg/k8sutil.Reconcile\n\t/go/src/github.com/banzaicloud/istio-operator/pkg/k8sutil/resource.go:153\ngithub.com/banzaicloud/istio-operator/pkg/resources/istiod.(*Reconciler).Reconcile\n\t/go/src/github.com/banzaicloud/istio-operator/pkg/resources/istiod/istiod.go:113\ngithub.com/banzaicloud/istio-operator/pkg/controller/istio.(*ReconcileIstio).reconcile\n\t/go/src/github.com/banzaicloud/istio-operator/pkg/controller/istio/istio_controller.go:355\ngithub.com/banzaicloud/istio-operator/pkg/controller/istio.(*ReconcileIstio).Reconcile\n\t/go/src/github.com/banzaicloud/istio-operator/pkg/controller/istio/istio_controller.go:239\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nfailed to reconcile resource\ncould not reconcile istio"

EKS version: 21
istio version: 1.7.8

The pods was in looping shutdown less than 2 minutes: "failed to reconcile resource could not reconcile istio"

Troubleshooting: We change the HPA config reboot all the pods disable and enable the ingress-gateway / istio-gateway

Could you please, help me about it?

Laci21 commented 2 years ago

Hi!

What's your Istio operator version? How did you enable an ingressgateway, through the Istio CR or the MeshGateway CR? Could you pls share your Istio and/or MeshGateway CR yamls as well?

Thanks!

stephanie-tavares commented 2 years ago

@Laci21 hello,

istio version: 1.7.8, using CR.

I've attached the file here because it’s too large:

istio-operator.yml.zip

Laci21 commented 2 years ago

Hi,

I'm looking for the Istio operator's version, not the Istio version. So docker image version of the istio-operator pod.

CR status looks good, ingress gw pods are up and running fine?

stephanie-tavares commented 2 years ago

Sure,

Istio operator's version: /banzaicloud/istio-operator:0.7.8

At the moment yes, are running fine. I'm try to understand what's happening.

Laci21 commented 2 years ago

Based on the error log it seems to be related to istiod deployment/pods. Please check that the istiod deployment and pods are up successfully as they should when you see this error.

There should be a program argument as well for the Istio operator to enable debug logging. With that you might see more (devel-mode=true maybe).