hashicorp / terraform-cloud-operator

Kubernetes Operator allows managing HCP Terraform resources via Kubernetes Custom Resources.
https://developer.hashicorp.com/terraform/cloud-docs
Mozilla Public License 2.0
114 stars 27 forks source link

"unable to decode an event from the watch" errors during pod termination #428

Open nabadger opened 4 weeks ago

nabadger commented 4 weeks ago

Operator Version, Kind and Kubernetes Version

YAML Manifest File

N/A

Output Log

2024-06-10 16:28:06    W0610 15:28:06.232834       1 reflector.go:462] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-10 16:37:37    W0610 15:37:37.304037       1 reflector.go:462] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: watch of *v1alpha2.Module ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-10 16:39:32    W0610 15:39:32.458770       1 reflector.go:462] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-10 16:39:32    W0610 15:39:32.458856       1 reflector.go:462] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-10 16:39:32    W0610 15:39:32.458910       1 reflector.go:462] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: watch of *v1alpha2.AgentPool ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-10 16:39:32    W0610 15:39:32.458946       1 reflector.go:462] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: watch of *v1alpha2.Module ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-10 16:39:32    W0610 15:39:32.459014       1 reflector.go:462] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: watch of *v1alpha2.Workspace ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-10 16:39:32    W0610 15:39:32.459064       1 reflector.go:462] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: watch of *v1alpha2.Project ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding

Output of relevant kubectl commands

N/A

Steps To Reproduce

I believe this only happens when replicas are set to 1 - so in the helm-chart we can just use

replicaCount: 1

When the pod is terminated, the errors are produced.

With only 1 pod, we don't have leader-election support. I'm fairly sure this error does not show up with >1 pods.

However, we still think running operators with 1 pod is fairly common.

Expected Behavior

The operator should terminate gracefully with no errors.

Actual Behavior

The operator terminates with errors. I have not confirmed if this happens when leader-election is used (>1 replica). I don't recall seeing these errors from memory though.

Additional Context

N/A

References

Community Note