search

StackStorm / stackstorm-k8s

K8s Helm Chart that codifies StackStorm (aka "IFTTT for Ops" https://stackstorm.com/) Highly Availability fleet as a simple to use reproducible infrastructure-as-code app
https://helm.stackstorm.com/
Apache License 2.0
105 stars 107 forks source link

failed to install stackstorm-ha on kubernetes using helm #364

Open acharbha opened 1 year ago

acharbha commented 1 year ago

More detailed information about failure:

PS C:\Users\user\repo> helm install stackstorm-ha .\stackstorm-ha-0.110.0.tgz --namespace=myspace --debug
install.go:194: [debug] Original chart version: ""
install.go:211: [debug] CHART PATH: C:\Users\user\repo\stackstorm-ha-0.110.0.tgz

client.go:133: [debug] creating 49 resource(s)
client.go:477: [debug] Starting delete for "stackstorm-ha-job-st2-apikey-load" Job
client.go:481: [debug] Ignoring delete failure for "stackstorm-ha-job-st2-apikey-load" batch/v1, Kind=Job: jobs.batch "stackstorm-ha-job-st2-apikey-load" not found
client.go:133: [debug] creating 1 resource(s)
client.go:703: [debug] Watching for changes to Job stackstorm-ha-job-st2-apikey-load with timeout of 5m0s
client.go:731: [debug] Add/Modify event for stackstorm-ha-job-st2-apikey-load: ADDED
client.go:770: [debug] stackstorm-ha-job-st2-apikey-load: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed post-install: timed out waiting for the condition
helm.go:84: [debug] failed post-install: timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
        helm.sh/helm/v3/cmd/helm/install.go:141
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.6.1/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.6.1/command.go:968
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1571
PS C:\Users\user\repo>
helm.exe version
version.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.18.10"}

kubectl.exe version --short
Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.20.15
PS C:\Users\user> .\kubectl.exe get all -n myspace
NAME                                                    READY   STATUS             RESTARTS   AGE
pod/stackstorm-ha-mongodb-0                             1/1     Running            0          8m53s
pod/stackstorm-ha-rabbitmq-0                            1/1     Running            0          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-6p46k      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-9bpvl      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-dlfks      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-hgjt2      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-z9jw2      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2api-767d4f5847-55hpd               0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2api-767d4f5847-kx7nl               0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2auth-578c967cf8-r88lr              0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2rulesengine-5f445b4d98-2n4hz       0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2rulesengine-5f445b4d98-4qvmf       0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2scheduler-ddb4b77b7-6xzsm          0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2scheduler-ddb4b77b7-ndbr7          0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2sensorcontainer-657cb59f7b-rt4c7   0/1     CrashLoopBackOff   6          8m54s
pod/stackstorm-ha-st2timersengine-6dcdbbc94c-nxf27      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2web-64644cf4c4-dtqmj               1/1     Running            0          8m54s
pod/stackstorm-ha-st2web-64644cf4c4-k669n               1/1     Running            0          8m54s
pod/stackstorm-ha-st2workflowengine-7d59b7c5c7-97lww    0/1     CrashLoopBackOff   6          8m54s
pod/stackstorm-ha-st2workflowengine-7d59b7c5c7-qw5wg    0/1     CrashLoopBackOff   6          8m54s

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
service/stackstorm-ha-mongodb-headless    ClusterIP   None             <none>        27017/TCP                               8m55s
service/stackstorm-ha-rabbitmq            ClusterIP   XX.XX.14.95     <none>        5672/TCP,4369/TCP,25672/TCP,15672/TCP   8m55s
service/stackstorm-ha-rabbitmq-headless   ClusterIP   None             <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   8m55s
service/stackstorm-ha-redis               ClusterIP   XX.XX.232.114   <none>        6379/TCP,26379/TCP                      8m55s
service/stackstorm-ha-redis-headless      ClusterIP   None             <none>        6379/TCP,26379/TCP                      8m55s
service/stackstorm-ha-st2api              ClusterIP   XX.XX.192.205   <none>        9101/TCP                                8m55s
service/stackstorm-ha-st2auth             ClusterIP   XX.XX.163.153   <none>        9100/TCP                                8m55s
service/stackstorm-ha-st2stream           ClusterIP   XX.XX.104.135   <none>        9102/TCP                                8m55s
service/stackstorm-ha-st2web              NodePort    XX.XX.83.234    <none>        80:32282/TCP                            8m55s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/stackstorm-ha-st2actionrunner       0/5     5            0           8m55s
deployment.apps/stackstorm-ha-st2api                0/2     2            0           8m54s
deployment.apps/stackstorm-ha-st2auth               0/2     1            0           8m54s
deployment.apps/stackstorm-ha-st2client             0/1     0            0           8m54s
deployment.apps/stackstorm-ha-st2garbagecollector   0/1     0            0           8m54s
deployment.apps/stackstorm-ha-st2notifier           0/2     0            0           8m54s
deployment.apps/stackstorm-ha-st2rulesengine        0/2     2            0           8m54s
deployment.apps/stackstorm-ha-st2scheduler          0/2     2            0           8m54s
deployment.apps/stackstorm-ha-st2sensorcontainer    0/1     1            0           8m55s
deployment.apps/stackstorm-ha-st2stream             0/2     0            0           8m54s
deployment.apps/stackstorm-ha-st2timersengine       0/1     1            0           8m54s
deployment.apps/stackstorm-ha-st2web                2/2     2            2           8m55s
deployment.apps/stackstorm-ha-st2workflowengine     0/2     2            0           8m55s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/stackstorm-ha-st2actionrunner-5df8dfb44f      5         5         0       8m55s
replicaset.apps/stackstorm-ha-st2api-767d4f5847               2         2         0       8m54s
replicaset.apps/stackstorm-ha-st2auth-578c967cf8              2         1         0       8m54s
replicaset.apps/stackstorm-ha-st2client-c7d56bf9c             1         0         0       8m54s
replicaset.apps/stackstorm-ha-st2garbagecollector-bf8f5bbd6   1         0         0       8m54s
replicaset.apps/stackstorm-ha-st2notifier-76d676748c          2         0         0       8m54s
replicaset.apps/stackstorm-ha-st2rulesengine-5f445b4d98       2         2         0       8m54s
replicaset.apps/stackstorm-ha-st2scheduler-ddb4b77b7          2         2         0       8m54s
replicaset.apps/stackstorm-ha-st2sensorcontainer-657cb59f7b   1         1         0       8m55s
replicaset.apps/stackstorm-ha-st2stream-6b96d4f4              2         0         0       8m54s
replicaset.apps/stackstorm-ha-st2timersengine-6dcdbbc94c      1         1         0       8m54s
replicaset.apps/stackstorm-ha-st2web-64644cf4c4               2         2         2       8m55s
replicaset.apps/stackstorm-ha-st2workflowengine-7d59b7c5c7    2         2         0       8m55s

NAME                                        READY   AGE
statefulset.apps/stackstorm-ha-mongodb      0/3     8m54s
statefulset.apps/stackstorm-ha-rabbitmq     0/3     8m54s
statefulset.apps/stackstorm-ha-redis-node   0/3     8m54s

NAME                                          COMPLETIONS   DURATION   AGE
job.batch/stackstorm-ha-job-st2-apikey-load   0/1                      8m53s
PS C:\Users\user>

helm list -n myspace
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS  CHART                   APP VERSION
stackstorm-ha   myspace       1               2023-03-16 22:23:17.5061257 +0530 IST   failed  stackstorm-ha-0.110.0   3.8
arm4b commented 1 year ago

The problem is with the backends not starting:

NAME                                        READY   AGE
statefulset.apps/stackstorm-ha-mongodb      0/3     8m54s
statefulset.apps/stackstorm-ha-rabbitmq     0/3     8m54s
statefulset.apps/stackstorm-ha-redis-node   0/3     8m54s

StackStorm HA relies on external helm charts for MongoDB, RabbitMQ, Redis which are used by the st2 cluster.

You'll need to debug those and find the reason why they're not starting. I'd recommend looking into a storage as that was a frequent issue with the K8s cluster setup.

Once they start normally, stackstorm will start too.