search

timescale / helm-charts

Configuration and Documentation to run TimescaleDB in your Kubernetes cluster
Apache License 2.0
263 stars 223 forks source link

Upgrade on Helm 3.3.0 leads to: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade failed: jobs.batch already exists #185

Closed ayeks closed 1 year ago

ayeks commented 4 years ago

Hey everyone,

when I run an helm upgrade on the timescale-db single chart with the latest Helm 3.3.0, it fails with the following error:

helm upgrade -v 10 -f config_timescaledb-generic.yaml timescaledb-generic charts/timescaledb-single/
..
Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade timescaledb-single/templates/job-update-patroni.yaml failed: jobs.batch "timescaledb-generic-patroni-w8" already exists

I downgraded to Helm 3.2.4 which fixed the issue. Unfortunately I have no idea where the problem originates from.

Best regards Lars

Details

My config:

# Config for timescaledb-generic deployment
# Overwrites charts/timescaledb-single/values.yaml
loadBalancer:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

persistentVolumes:
  data:
    size: 100Gi
  wal:
    size: 20Gi

Details Error on Helm 3.3.0

Details Jenkins log:

I0724 06:52:12.462544    6849 round_trippers.go:420] GET https://XXXXXXXXXXXXXXXXx.yl4.eu-central-1.eks.amazonaws.com/apis/batch/v1/namespaces/default/jobs/timescaledb-generic-patroni-w8
I0724 06:52:12.462557    6849 round_trippers.go:427] Request Headers:
I0724 06:52:12.462565    6849 round_trippers.go:431]     Accept: application/json
I0724 06:52:12.465295    6849 round_trippers.go:446] Response Status: 404 Not Found in 2 milliseconds
I0724 06:52:12.465308    6849 round_trippers.go:449] Response Headers:
I0724 06:52:12.465329    6849 round_trippers.go:452]     Audit-Id: 0f54bf55-1557-4de1-852a-58090e5a9cae
I0724 06:52:12.465335    6849 round_trippers.go:452]     Cache-Control: no-cache, private
I0724 06:52:12.465340    6849 round_trippers.go:452]     Content-Type: application/json
I0724 06:52:12.465345    6849 round_trippers.go:452]     Content-Length: 250
I0724 06:52:12.465351    6849 round_trippers.go:452]     Date: Fri, 24 Jul 2020 06:52:12 GMT
I0724 06:52:12.465368    6849 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"timescaledb-generic-patroni-w8\" not found","reason":"NotFound","details":{"name":"timescaledb-generic-patroni-w8","group":"batch","kind":"jobs"},"code":404}
I0724 06:52:12.465542    6849 request.go:1068] Request Body: {"apiVersion":"batch/v1","kind":"Job","metadata":{"annotations":{"helm.sh/hook":"post-upgrade","helm.sh/hook-delete-policy":"hook-succeeded","meta.helm.sh/release-name":"timescaledb-generic","meta.helm.sh/release-namespace":"default"},"labels":{"app":"timescaledb-generic","app.kubernetes.io/managed-by":"Helm","chart":"timescaledb-single-0.6.2","cluster-name":"timescaledb-generic","heritage":"Helm","release":"timescaledb-generic"},"name":"timescaledb-generic-patroni-w8","namespace":"default"},"spec":{"activeDeadlineSeconds":120,"template":{"metadata":{"labels":{"app":"timescaledb-generic","chart":"timescaledb-single-0.6.2","heritage":"Helm","release":"timescaledb-generic"}},"spec":{"containers":[{"args":["-c","/usr/bin/curl --connect-timeout 30 --include --request PATCH --data \\\n\"{\\\"loop_wait\\\":10,\\\"maximum_lag_on_failover\\\":33554432,\\\"postgresql\\\":{\\\"parameters\\\":{\\\"archive_command\\\":\\\"/etc/timescaledb/scripts/pgbackrest_archive.sh %p\\\",\\\"archive_mode\\\":\\\"on\\\",\\\"archive_ti [truncated 1487 chars]
I0724 06:52:12.465584    6849 round_trippers.go:420] POST https://XXXXXXXXXXXXXXXXx.yl4.eu-central-1.eks.amazonaws.com/apis/batch/v1/namespaces/default/jobs
I0724 06:52:12.465592    6849 round_trippers.go:427] Request Headers:
I0724 06:52:12.465599    6849 round_trippers.go:431]     Accept: application/json
I0724 06:52:12.465605    6849 round_trippers.go:431]     Content-Type: application/json
I0724 06:52:12.470389    6849 round_trippers.go:446] Response Status: 201 Created in 4 milliseconds
I0724 06:52:12.470403    6849 round_trippers.go:449] Response Headers:
I0724 06:52:12.470416    6849 round_trippers.go:452]     Content-Type: application/json
I0724 06:52:12.470422    6849 round_trippers.go:452]     Content-Length: 3224
I0724 06:52:12.470427    6849 round_trippers.go:452]     Date: Fri, 24 Jul 2020 06:52:12 GMT
I0724 06:52:12.470433    6849 round_trippers.go:452]     Audit-Id: a5d06a78-af37-47c5-af54-88ae36bec229
I0724 06:52:12.470438    6849 round_trippers.go:452]     Cache-Control: no-cache, private
I0724 06:52:12.470471    6849 request.go:1068] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"timescaledb-generic-patroni-w8","namespace":"default","selfLink":"/apis/batch/v1/namespaces/default/jobs/timescaledb-generic-patroni-w8","uid":"7f907799-5800-4d9c-8aed-14ad05f5b5c5","resourceVersion":"1560480","creationTimestamp":"2020-07-24T06:52:12Z","labels":{"app":"timescaledb-generic","app.kubernetes.io/managed-by":"Helm","chart":"timescaledb-single-0.6.2","cluster-name":"timescaledb-generic","heritage":"Helm","release":"timescaledb-generic"},"annotations":{"helm.sh/hook":"post-upgrade","helm.sh/hook-delete-policy":"hook-succeeded","meta.helm.sh/release-name":"timescaledb-generic","meta.helm.sh/release-namespace":"default"}},"spec":{"parallelism":1,"completions":1,"activeDeadlineSeconds":120,"backoffLimit":6,"selector":{"matchLabels":{"controller-uid":"7f907799-5800-4d9c-8aed-14ad05f5b5c5"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"timescaledb-generic","chart":"timescaledb-single-0.6.2","controller-uid":"7f907799-5800-4d9c [truncated 2200 chars]
I0724 06:52:12.513112    6849 round_trippers.go:420] POST https://XXXXXXXXXXXXXXXXx.yl4.eu-central-1.eks.amazonaws.com/apis/batch/v1/namespaces/default/jobs
I0724 06:52:12.513123    6849 round_trippers.go:427] Request Headers:
I0724 06:52:12.513130    6849 round_trippers.go:431]     Accept: application/json
I0724 06:52:12.513137    6849 round_trippers.go:431]     Content-Type: application/json
I0724 06:52:12.521116    6849 round_trippers.go:446] Response Status: 409 Conflict in 7 milliseconds
I0724 06:52:12.521148    6849 round_trippers.go:449] Response Headers:
I0724 06:52:12.521155    6849 round_trippers.go:452]     Content-Length: 260
I0724 06:52:12.521161    6849 round_trippers.go:452]     Date: Fri, 24 Jul 2020 06:52:12 GMT
I0724 06:52:12.521166    6849 round_trippers.go:452]     Audit-Id: 82e4f89a-7412-4e32-9a9e-b4508e55ffeb
I0724 06:52:12.521171    6849 round_trippers.go:452]     Cache-Control: no-cache, private
I0724 06:52:12.521177    6849 round_trippers.go:452]     Content-Type: application/json
I0724 06:52:12.521212    6849 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"timescaledb-generic-patroni-w8\" already exists","reason":"AlreadyExists","details":{"name":"timescaledb-generic-patroni-w8","group":"batch","kind":"jobs"},"code":409}
I0724 06:52:12.536359    6849 request.go:1068] Request Body: {"kind":"Secret","apiVersion":"v1","metadata":{"name":"sh.helm.release.v1.timescaledb-generic.v24","creationTimestamp":null,"labels":{"modifiedAt":"1595573532","name":"timescaledb-generic","owner":"helm","status":"failed","version":"24"}},"data":{"release":"SDRzSUFBQUFBQUFDLyt6OWVYT2p5UFluakw4VmgzL3p4MjlteW5VQldlNmlJanJpRVVnZ3NJUU5FdXYweERmWURFakowZ1l0Nk01OTcwOWtKaUJBeUV0WDlYMXVUUGNmSFYyV1JKTEx5Yk9mei9ubmJXTEgvdTMzMnlLSy9keTFnZTg1ZDRHZitLK1JlL3ZsTmtwZTB0dnYvN3g5aVY3ejRyODhQd05wNlh1MzMyOHBnaUx1aUYvdUtHSk5FdC9KMGZkNzRpdE4wTCtNUmcvM0Q5YnRsMXRnWDNuZ2ZrMDhmQjlUMzBueTZ6ZmlnYVRIeEMvZjRBT2VEL3dDL1JUOWtidXZVVlpFYVhMNy9WYk5nbGZiODI5K0c1cmxiN2MzTDNZRWZPLzdUWmJteGQydStuR1lwdHU4K2VwZ3Z5WlJFbnkvbWFmcHR2dkQ5cGg1bEFUQS8wZmh4eG13Q3ovL3h5WjE3bmFaWnhmK1hXWVhyMmtTZlMzdEdEVGpibEluLytyWWhSc096NjUrNnU3dzdiZmJHeHU4K3JaWDN2akhLQy95MnkrM2VXRVh1L3oyK3kwZTcvYkxiWklXUHZ6Z3QyUmRqelpsYmx3N3VYSDhHOXQxL1R6M3ZadDlaTjlrNld0eE03NGZVVGRwY2xPRS9zMUxDa0I2aUpMZ1ppcXRidUM1M3J5OHB2SE5JU3JDS0xrcDA5M3JqUXQyZWVHL2Z2OHRHWmpzVjg5L3NYZWcrSnJ2M2EvVkw3K0MxTF [truncated 112250 chars]

...

Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade timescaledb-single/templates/job-update-patroni.yaml failed: jobs.batch "timescaledb-generic-patroni-w8" already exists

The jobs which are triggered by helm upgrade aren't removed afterwards if it fails:

$ kubectl get jobs
NAME                             COMPLETIONS   DURATION   AGE
timescaledb-generic-patroni-2q   1/1           3s         42m
timescaledb-generic-patroni-3o   1/1           3s         121m
timescaledb-generic-patroni-3s   1/1           3s         74m
timescaledb-generic-patroni-49   1/1           3s         2d23h
timescaledb-generic-patroni-4p   1/1           3s         98m
timescaledb-generic-patroni-4s   1/1           5s         2d23h
timescaledb-generic-patroni-5g   1/1           3s         2d23h
timescaledb-generic-patroni-99   1/1           4s         2d23h
timescaledb-generic-patroni-d2   1/1           3s         17m
timescaledb-generic-patroni-r4   1/1           3s         84m
timescaledb-generic-patroni-w8   1/1           3s         14m
feikesteenbergen commented 4 years ago

I think these issues are related:

https://github.com/timescale/timescaledb-kubernetes/issues/106 https://github.com/timescale/timescaledb-kubernetes/issues/141

Which are fixed in: https://github.com/timescale/timescaledb-kubernetes/pull/107

Which was released with version 0.6.1

However, you first need to get rid of these jobs before you can proceed, you can do it manually or do something like:

kubectl get job,pod -l cluster-name -o name | grep '-patroni-' | xargs kubectl delete

To delete the jobs/pods that are related to Patroni