fluxcd / helm-controller

The GitOps Toolkit Helm reconciler, for declarative Helming
https://fluxcd.io
Apache License 2.0
408 stars 163 forks source link

install retries exhausted but the pod is running ok #701

Closed smartkuk closed 1 year ago

smartkuk commented 1 year ago

Used Google translate

hi I am currently using the flux v1 version and want to change to the v2 version, so I am researching it. While trying to install and deploy to a kind cluster in my local WSL Ubuntu environment, a strange phenomenon occurred, so I made an inquiry here.

The status of the tomcat Pod resource created through the HelmRelease resource is normal, but why does the status of the HelmRelease resource become an error after waiting for about 5 minutes? I googled a lot and followed all the steps related to "retries exhausted". But nothing worked. Shouldn't the contents that are uploaded from Get Started or Installation that you guys have made work at least to build trust in the component?

Below is kind cluster configuration.

---
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
name: taiga-1026
nodes:
- role: control-plane
  image: kindest/node:v1.24.13@sha256:cea86276e698af043af20143f4bf0509e730ec34ed3b7fa790cc0bea091bc5dd
- role: worker
  image: kindest/node:v1.24.13@sha256:cea86276e698af043af20143f4bf0509e730ec34ed3b7fa790cc0bea091bc5dd

Below is the client version I am using.

flux -v
flux version 2.0.0-rc.5

I registered the Helm chart and wanted to distribute the chart, so I set it up as follows.

flux install \
  --network-policy=false \
  --watch-all-namespaces=true \
  --namespace=flux-system \
  --log-level debug \
  --components=source-controller,helm-controller

And the nginx chart was installed with the HelmRepository setting completed normally.

flux create helmrelease nginx-testing-1 \
  --namespace=$NS \
  --target-namespace=$NS \
  --create-target-namespace \
  --source=HelmRepository/bxcp-system-common.bxcp-system \
  --chart=tomcat \
  --chart-version="10.9.2" \
  --interval=10s \
  --chart-interval=10s
✚ generating HelmRelease
► applying HelmRelease
✔ HelmRelease created
◎ waiting for HelmRelease reconciliation
✗ client rate limiter Wait returned an error: context deadline exceeded

After waiting for about 5 minutes, the nginx-testing-1 HelmRelease resource was found in the following state. The problem is that the Pod resource status is normal, but I don't know why the HelmRelease resource is abnormal.

The following is the result of checking the status with the kubectl command. Some information has been covered with garbage information.

Every 2.0s: kubectl get helmrepo -A; echo;echo; kubectl get hr -A; echo;echo; kubectl get pods -A                                                      NB-21042711: Fri Jun 16 08:41:27 2023

NAMESPACE     NAME                 URL                                                                                                              AGE   READY   STATUS
bxcp-system   bxcp-system-common   https://xxxxxxxxxx:0000/chartrepo/bxcp-system-common   29m   True    stored artifact: revision
'sha256:ea7575a7761511264744c90f5e0f63c024ca822c4714a73ae5bb003adc1d09c0'

NAMESPACE   NAME              AGE   READY   STATUS
default     nginx-testing-1   25m   False   install retries exhausted

NAMESPACE            NAME                                               READY   STATUS    RESTARTS   AGE
default              default-nginx-testing-1-tomcat-86c9545c98-mhh72    1/1     Running   0          25m
flux-system          helm-controller-b4ffbcbf-hgqfb                     1/1     Running   0          29m
flux-system          source-controller-659b55846c-wk2c6                 1/1     Running   0          29m
kube-system          coredns-57575c5f89-t6rx9                           1/1     Running   0          30m
kube-system          coredns-57575c5f89-xpr78                           1/1     Running   0          30m
kube-system          etcd-taiga-1026-control-plane                      1/1     Running   0          30m
kube-system          kindnet-774lg                                      1/1     Running   0          29m
kube-system          kindnet-jztlr                                      1/1     Running   0          30m
kube-system          kube-apiserver-taiga-1026-control-plane            1/1     Running   0          30m
kube-system          kube-controller-manager-taiga-1026-control-plane   1/1     Running   0          30m
kube-system          kube-proxy-fx6cf                                   1/1     Running   0          29m
kube-system          kube-proxy-gfqxm                                   1/1     Running   0          30m
kube-system          kube-scheduler-taiga-1026-control-plane            1/1     Running   0          30m
local-path-storage   local-path-provisioner-c49b7b56f-jdhgc             1/1     Running   0          30m

Here is the helm controller log:

{"level":"info","ts":"2023-06-15T23:11:58.719Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2023-06-15T23:11:58.720Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2023-06-15T23:11:58.720Z","msg":"Starting server","kind":"health probe","addr":"[::]:9440"}
{"level":"info","ts":"2023-06-15T23:11:58.821Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2023-06-15T23:11:58.821Z","logger":"runtime","msg":"attempting to acquire leader lease flux-system/helm-controller-leader-election...\n"}
{"level":"info","ts":"2023-06-15T23:11:58.828Z","logger":"runtime","msg":"successfully acquired lease flux-system/helm-controller-leader-election\n"}
{"level":"debug","ts":"2023-06-15T23:11:58.828Z","logger":"events","msg":"helm-controller-b4ffbcbf-hgqfb_e9ac28cd-408e-44a8-ac22-d90f7d62c611 became leader","type":"Normal","object":{"kind":"Lease","namespace":"flux-system","name":"helm-controller-leader-election","uid":"c207f0ff-ab50-4c6f-9028-7bfdf293c7bf","apiVersion":"coordination.k8s.io/v1","resourceVersion":"627"},"reason":"LeaderElection"}
{"level":"info","ts":"2023-06-15T23:11:58.828Z","msg":"Starting EventSource","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","source":"kind source: *v2beta1.HelmRelease"}
{"level":"info","ts":"2023-06-15T23:11:58.828Z","msg":"Starting EventSource","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","source":"kind source: *v1beta2.HelmChart"}
{"level":"info","ts":"2023-06-15T23:11:58.828Z","msg":"Starting Controller","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease"}
{"level":"info","ts":"2023-06-15T23:11:58.930Z","msg":"Starting workers","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","worker count":4}
{"level":"info","ts":"2023-06-15T23:16:27.256Z","msg":"HelmChart 'bxcp-system/default-nginx-testing-1' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"0a81762c-f7e4-4157-9cee-6d6ca0498b69"}
{"level":"debug","ts":"2023-06-15T23:16:27.256Z","logger":"events","msg":"HelmChart 'bxcp-system/default-nginx-testing-1' is not ready","type":"Normal","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1104"},"reason":"info"}
{"level":"info","ts":"2023-06-15T23:16:27.268Z","msg":"reconcilation finished in 38.1457ms, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"0a81762c-f7e4-4157-9cee-6d6ca0498b69"}
{"level":"debug","ts":"2023-06-15T23:16:28.174Z","logger":"events","msg":"Helm install has started","type":"Normal","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1109"},"reason":"info"}
{"level":"debug","ts":"2023-06-15T23:21:28.873Z","logger":"events","msg":"Helm install failed: context deadline exceeded\n\nLast Helm logs:\n\n","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1109"},"reason":"error"}
{"level":"debug","ts":"2023-06-15T23:21:28.873Z","logger":"events","msg":"reconciliation failed: Helm install failed: context deadline exceeded","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1109"},"reason":"error"}
{"level":"info","ts":"2023-06-15T23:21:28.886Z","msg":"reconcilation finished in 5m0.726131s, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"4c28e2ad-2e2d-4e64-8ca5-a46b89086b1d"}
{"level":"error","ts":"2023-06-15T23:21:28.886Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"4c28e2ad-2e2d-4e64-8ca5-a46b89086b1d","error":"Helm install failed: context deadline exceeded","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"debug","ts":"2023-06-15T23:21:28.900Z","logger":"events","msg":"reconciliation failed: install retries exhausted","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1715"},"reason":"error"}
{"level":"info","ts":"2023-06-15T23:21:28.930Z","msg":"reconcilation finished in 44.1972ms, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"204fabd9-977b-4be7-8574-4c557b58a4c5"}
{"level":"error","ts":"2023-06-15T23:21:28.930Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"204fabd9-977b-4be7-8574-4c557b58a4c5","error":"install retries exhausted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"debug","ts":"2023-06-15T23:21:29.644Z","logger":"events","msg":"reconciliation failed: install retries exhausted","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1718"},"reason":"error"}
{"level":"info","ts":"2023-06-15T23:21:29.654Z","msg":"reconcilation finished in 17.8927ms, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"5614902c-69a9-4c42-8c19-7c14b77f1dee"}
{"level":"error","ts":"2023-06-15T23:21:29.654Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"5614902c-69a9-4c42-8c19-7c14b77f1dee","error":"install retries exhausted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"debug","ts":"2023-06-15T23:21:32.667Z","logger":"events","msg":"reconciliation failed: install retries exhausted","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1720"},"reason":"error"}
{"level":"info","ts":"2023-06-15T23:21:32.677Z","msg":"reconcilation finished in 22.4546ms, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"bc660ef4-1e40-42d4-a7d8-4bf92f371d1f"}
{"level":"error","ts":"2023-06-15T23:21:32.677Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"bc660ef4-1e40-42d4-a7d8-4bf92f371d1f","error":"install retries exhausted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"debug","ts":"2023-06-15T23:21:38.687Z","logger":"events","msg":"reconciliation failed: install retries exhausted","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1723"},"reason":"error"}
{"level":"info","ts":"2023-06-15T23:21:38.698Z","msg":"reconcilation finished in 19.9074ms, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"7cc001b6-551a-457e-9983-eff6cbfa904e"}
{"level":"error","ts":"2023-06-15T23:21:38.698Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"7cc001b6-551a-457e-9983-eff6cbfa904e","error":"install retries exhausted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"debug","ts":"2023-06-15T23:21:50.720Z","logger":"events","msg":"reconciliation failed: install retries exhausted","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1738"},"reason":"error"}
{"level":"info","ts":"2023-06-15T23:21:50.737Z","msg":"reconcilation finished in 36.8876ms, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"c5092ff0-885c-4e2e-bcad-feee3b3e8b75"}
{"level":"error","ts":"2023-06-15T23:21:50.737Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"c5092ff0-885c-4e2e-bcad-feee3b3e8b75","error":"install retries exhausted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"debug","ts":"2023-06-15T23:22:14.748Z","logger":"events","msg":"reconciliation failed: install retries exhausted","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1760"},"reason":"error"}
{"level":"info","ts":"2023-06-15T23:22:14.788Z","msg":"reconcilation finished in 50.2202ms, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"43785a61-7a45-4466-bb05-d7c569fdb0ff"}
{"level":"error","ts":"2023-06-15T23:22:14.788Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"43785a61-7a45-4466-bb05-d7c569fdb0ff","error":"install retries exhausted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"debug","ts":"2023-06-15T23:23:02.803Z","logger":"events","msg":"reconciliation failed: install retries exhausted","type":"Warning","object":{"kind":"HelmRelease","namespace":"default","name":"nginx-testing-1","uid":"ba9618a1-fc7c-4800-ab31-a35489bf5500","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","resourceVersion":"1805"},"reason":"error"}
{"level":"info","ts":"2023-06-15T23:23:02.818Z","msg":"reconcilation finished in 28.1489ms, next run in 10s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"d8b16583-71d8-42ce-9615-f1b3a826fa2a"}
{"level":"error","ts":"2023-06-15T23:23:02.818Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"nginx-testing-1","namespace":"default"},"namespace":"default","name":"nginx-testing-1","reconcileID":"d8b16583-71d8-42ce-9615-f1b3a826fa2a","error":"install retries exhausted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

Below is the tomcat pod log. You might be confused because nginx is included in the name, but it is tomcat.

kubectl logs -f default-nginx-testing-1-tomcat-86c9545c98-mhh72
tomcat 23:16:54.24
tomcat 23:16:54.24 Welcome to the Bitnami tomcat container
tomcat 23:16:54.24 Subscribe to project updates by watching https://github.com/bitnami/containers
tomcat 23:16:54.24 Submit issues and feature requests at https://github.com/bitnami/containers/issues
tomcat 23:16:54.24
tomcat 23:16:54.24 INFO  ==> ** Starting tomcat setup **
tomcat 23:16:54.26 INFO  ==> Configuring port numbers
tomcat 23:16:54.26 INFO  ==> Creating Tomcat user
tomcat 23:16:54.27 INFO  ==> Ensuring Tomcat directories exist
tomcat 23:16:54.27 INFO  ==> Deploying Tomcat from scratch
tomcat 23:16:54.32 INFO  ==> ** tomcat setup finished! **

tomcat 23:16:54.32 INFO  ==> ** Starting Tomcat **
15-Jun-2023 23:16:54.708 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/10.1.9
15-Jun-2023 23:16:54.712 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          May 9 2023 14:30:05 UTC
15-Jun-2023 23:16:54.712 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.1.9.0
15-Jun-2023 23:16:54.712 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
15-Jun-2023 23:16:54.713 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.15.49-linuxkit
15-Jun-2023 23:16:54.713 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
15-Jun-2023 23:16:54.713 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /opt/bitnami/java
15-Jun-2023 23:16:54.713 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           17.0.7+7-LTS
15-Jun-2023 23:16:54.713 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            BellSoft
15-Jun-2023 23:16:54.713 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /opt/bitnami/tomcat
15-Jun-2023 23:16:54.713 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /opt/bitnami/tomcat
15-Jun-2023 23:16:54.722 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/bitnami/tomcat/conf/logging.properties
15-Jun-2023 23:16:54.723 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
15-Jun-2023 23:16:54.723 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
15-Jun-2023 23:16:54.723 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseG1GC
15-Jun-2023 23:16:54.723 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfile.encoding=UTF-8
15-Jun-2023 23:16:54.723 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.net.preferIPv4Stack=true
15-Jun-2023 23:16:54.723 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.net.preferIPv4Addresses=true
15-Jun-2023 23:16:54.723 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Duser.home=/opt/bitnami/tomcat
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/bitnami/tomcat
15-Jun-2023 23:16:54.724 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/bitnami/tomcat
15-Jun-2023 23:16:54.729 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/bitnami/tomcat/temp
15-Jun-2023 23:16:54.731 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent An older version [1.2.37] of the Apache Tomcat Native library is installed, while Tomcat recommends a minimum version of [2.0.1]
15-Jun-2023 23:16:54.732 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.37] using APR version [1.7.4].
15-Jun-2023 23:16:54.733 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1n  15 Mar 2022]
15-Jun-2023 23:16:54.911 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
15-Jun-2023 23:16:54.927 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [381] milliseconds
15-Jun-2023 23:16:54.953 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
15-Jun-2023 23:16:54.953 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/10.1.9]
15-Jun-2023 23:16:54.959 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/bitnami/tomcat/webapps/manager]
15-Jun-2023 23:16:55.253 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
15-Jun-2023 23:16:55.294 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/bitnami/tomcat/webapps/manager] has finished in [336] ms
15-Jun-2023 23:16:55.294 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/bitnami/tomcat/webapps/host-manager]
15-Jun-2023 23:16:55.392 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
15-Jun-2023 23:16:55.394 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/bitnami/tomcat/webapps/host-manager] has finished in [100] ms
15-Jun-2023 23:16:55.394 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/bitnami/tomcat/webapps/docs]
15-Jun-2023 23:16:55.471 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
15-Jun-2023 23:16:55.472 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/bitnami/tomcat/webapps/docs] has finished in [78] ms
15-Jun-2023 23:16:55.473 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/bitnami/tomcat/webapps/ROOT]
15-Jun-2023 23:16:55.544 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
15-Jun-2023 23:16:55.545 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/bitnami/tomcat/webapps/ROOT] has finished in [72] ms
15-Jun-2023 23:16:55.545 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/bitnami/tomcat/webapps/examples]
15-Jun-2023 23:16:55.685 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
15-Jun-2023 23:16:55.707 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/bitnami/tomcat/webapps/examples] has finished in [162] ms
15-Jun-2023 23:16:55.709 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
15-Jun-2023 23:16:55.719 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [791] milliseconds
smartkuk commented 1 year ago

Additionally, if I use the helm command to install the chart in the same repository, the status looks correct. Some information has been covered with garbage information.

The storage under the harbor name is the same storage used above.

$ helm repo list
NAME                    URL
prometheus-community    https://prometheus-community.github.io/helm-charts
stable                  https://charts.helm.sh/stable
ingress-nginx           https://kubernetes.github.io/ingress-nginx
harbor                  https://xxxxxxxxxx:0000/chartrepo/bxcp-system-common
argo                    https://argoproj.github.io/argo-helm
member-harbor           https://yyyyyyyyyy:0000/chartrepo/bxcp-system-common
$ helm install taiga-1026 harbor/tomcat
NAME: taiga-1026
LAST DEPLOYED: Fri Jun 16 09:50:14 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: tomcat
CHART VERSION: 10.9.2
APP VERSION: 10.1.9

** Please be patient while the chart is being deployed **

1. Get the Tomcat URL by running:

  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace default -w taiga-1026-tomcat'

  export SERVICE_IP=$(kubectl get svc --namespace default taiga-1026-tomcat --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")
  echo "Tomcat URL:            http://$SERVICE_IP/"
  echo "Tomcat Management URL: http://$SERVICE_IP/manager"

2. Login with the following credentials

  echo Username: user
  echo Password: $(kubectl get secret --namespace default taiga-1026-tomcat -o jsonpath="{.data.tomcat-password}" | base64 -d)
$ helm history taiga-1026
REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION
1               Fri Jun 16 09:50:14 2023        deployed        tomcat-10.9.2   10.1.9          Install complete
hiddeco commented 1 year ago

Can you please try increasing the timeout configuration on the HelmRelease? 5 minutes matches the default time-out.

smartkuk commented 1 year ago

Can you please try increasing the timeout configuration on the HelmRelease? 5 minutes matches the default time-out.

Are you talking about timeout in the sub spec? I've already done that. But just in case, I tried again. But again, it didn't work out. Below is the spec.

# timeout 10m
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: tomcat-testing-1
  namespace: default
spec:
  timeout: 10m
  chart:
    spec:
      chart: tomcat
      interval: 10s
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: bxcp-system-common
        namespace: bxcp-system
      version: 10.9.2
  install:
    createNamespace: true
    remediation:
      retries: 10
  interval: 10s
  targetNamespace: default

Below is what I monitored with the watch command.

Every 2.0s: kubectl get helmrepo -A;echo;echo;kubectl get hr -A;echo;echo;helm list --all --all-namespaces;echo;echo;kubectl get pods -A;              NB-21042711: Mon Jun 19 08:27:31 2023

NAMESPACE     NAME                 URL                                                                                                              AGE   READY   STATUS
bxcp-system   bxcp-system-common   https://xxxxxxxxxx:0000/chartrepo/bxcp-system-common   13m   True    stored artifact: revision
'sha256:ea7575a7761511264744c90f5e0f63c024ca822c4714a73ae5bb003adc1d09c0'

NAMESPACE   NAME               AGE     READY     STATUS
default     tomcat-testing-1   9m26s   Unknown   Reconciliation in progress

NAME                            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
default-tomcat-testing-1        default         1               2023-06-18 23:18:05.9933536 +0000 UTC   pending-install tomcat-10.9.2   10.1.9

NAMESPACE            NAME                                               READY   STATUS    RESTARTS   AGE
default              default-tomcat-testing-1-5cf8d477d6-ktvjt          1/1     Running   0          9m26s
flux-system          helm-controller-b4ffbcbf-tfv96                     1/1     Running   0          14m
flux-system          source-controller-659b55846c-7jkzz                 1/1     Running   0          14m
kube-system          coredns-57575c5f89-hjc2r                           1/1     Running   0          14m
kube-system          coredns-57575c5f89-r95vc                           1/1     Running   0          14m
kube-system          etcd-taiga-1026-control-plane                      1/1     Running   0          14m
kube-system          kindnet-74z2x                                      1/1     Running   0          14m
kube-system          kindnet-f4m99                                      1/1     Running   0          14m
kube-system          kube-apiserver-taiga-1026-control-plane            1/1     Running   0          14m
kube-system          kube-controller-manager-taiga-1026-control-plane   1/1     Running   0          14m
kube-system          kube-proxy-fwktv                                   1/1     Running   0          14m
kube-system          kube-proxy-r5bbz                                   1/1     Running   0          14m
kube-system          kube-scheduler-taiga-1026-control-plane            1/1     Running   0          14m
local-path-storage   local-path-provisioner-c49b7b56f-vdw7f             1/1     Running   0          14m

At the point of 10 minutes, the JOB resource is automatically created and the status is changed to an error.

Every 2.0s: kubectl get helmrepo -A;echo;echo;kubectl get hr -A;echo;echo;helm list --all --all-namespaces;echo;echo;kubectl get pods -A;              NB-21042711: Mon Jun 19 08:28:35 2023

NAMESPACE     NAME                 URL                                                                                                              AGE   READY   STATUS
bxcp-system   bxcp-system-common   https://host-host-infra-lb-2828a07e11a75d4e.elb.ap-northeast-2.amazonaws.com:5443/chartrepo/bxcp-system-common   15m   True    stored artifact: revision
'sha256:ea7575a7761511264744c90f5e0f63c024ca822c4714a73ae5bb003adc1d09c0'

NAMESPACE   NAME               AGE   READY   STATUS
default     tomcat-testing-1   10m   False   Helm install failed: context deadline exceeded

NAME                            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
default-tomcat-testing-1        default         1               2023-06-18 23:28:08.4813081 +0000 UTC   pending-install tomcat-10.9.2   10.1.9

NAMESPACE            NAME                                               READY   STATUS    RESTARTS   AGE
default              default-tomcat-testing-1-5cf8d477d6-tbh22          0/1     Running   0          27s
flux-system          helm-controller-b4ffbcbf-tfv96                     1/1     Running   0          15m
flux-system          source-controller-659b55846c-7jkzz                 1/1     Running   0          15m
kube-system          coredns-57575c5f89-hjc2r                           1/1     Running   0          15m
kube-system          coredns-57575c5f89-r95vc                           1/1     Running   0          15m
kube-system          etcd-taiga-1026-control-plane                      1/1     Running   0          15m
kube-system          kindnet-74z2x                                      1/1     Running   0          15m
kube-system          kindnet-f4m99                                      1/1     Running   0          15m
kube-system          kube-apiserver-taiga-1026-control-plane            1/1     Running   0          15m
kube-system          kube-controller-manager-taiga-1026-control-plane   1/1     Running   0          15m
kube-system          kube-proxy-fwktv                                   1/1     Running   0          15m
kube-system          kube-proxy-r5bbz                                   1/1     Running   0          15m
kube-system          kube-scheduler-taiga-1026-control-plane            1/1     Running   0          15m
local-path-storage   local-path-provisioner-c49b7b56f-vdw7f             1/1     Running   0          15m
helm status default-tomcat-testing-1
NAME: default-tomcat-testing-1
LAST DEPLOYED: Sun Jun 18 23:28:08 2023
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: tomcat
CHART VERSION: 10.9.2
APP VERSION: 10.1.9

** Please be patient while the chart is being deployed **

1. Get the Tomcat URL by running:

  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace default -w default-tomcat-testing-1'

  export SERVICE_IP=$(kubectl get svc --namespace default default-tomcat-testing-1 --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")
  echo "Tomcat URL:            http://$SERVICE_IP/"
  echo "Tomcat Management URL: http://$SERVICE_IP/manager"

2. Login with the following credentials

  echo Username: user
  echo Password: $(kubectl get secret --namespace default default-tomcat-testing-1 -o jsonpath="{.data.tomcat-password}" | base64 -d)

I've mentioned it before, but I want you to know that I've tried all of the ways to do this by Googling.

hiddeco commented 1 year ago

Would you be able to share the chart with me? Can be via email, hidde@weave.works.

smartkuk commented 1 year ago

Would you be able to share the chart with me? Can be via email, hidde@weave.works.

OK I sent email, Subject is "[tomcat chart] hi this is my chart"

hiddeco commented 1 year ago

I tested the chart you provided, and I suspect it times out because the LoadBalancer does not get an external IP assigned.

$ kubectl get service
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx-testing-1-tomcat    LoadBalancer   10.96.210.40    <pending>     80:31829/TCP   13m

This is confirmed by changing the .spec of the HelmRelease to include:

spec:
  install:
    disableWait: true
  upgrade:
    disableWait: true

which yields a successful install.

smartkuk commented 1 year ago

nice to see the answer I also avoided the problem once through the settings you showed. However, what I'm curious about is that the result of installing with the install subcommand with the helm binary command and the result of installing with the HelmRelease resource are different.

If you install both with the same values.yaml spec, they should be in the same state, but they are not Is the reason for this result because the application of helmrelease resources is still in the development stage?

hiddeco commented 1 year ago

Helm would have given you the same behavior if you run helm install with the --wait flag. Which is the default the controller runs with, as we want to be sure the resources are successfully deployed by default.

smartkuk commented 1 year ago

Helm would have given you the same behavior if you run helm install with the --wait flag. Which is the default the controller runs with, as we want to be sure the resources are successfully deployed by default.

I didn't explore the options in detail I misunderstood. sorry Thank you so much for your sincere answer