search

jfrog / charts

JFrog official Helm Charts
https://jfrog.com/integration/helm-repository/
Apache License 2.0
259 stars 448 forks source link

Pre-upgrade migration hook terminates unsuccessfully #1890

Open dipop1 opened 5 months ago

dipop1 commented 5 months ago

Is this a request for help?:

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Version of Helm and Kubernetes:

Helm v3 Kubernetes v1.27.5

Which chart:

Jfrog Platform 1.18.0

Which product license (Enterprise/Pro/oss):

Enterprise Plus

JFrog support reference (if already raised with support team):

301924

What happened:

JFrog platform currently deployed on our k8s clusters is 10.17.0 (version of the official helm chart).

We're trying to upgrade it to v10.18.0, but we're experiencing issues with the rabbitmq instances. There are 2 rabbitmq replicas running on staging, and 3 on prod cluster at the moment.

The new rabbitmq image included in the v10.18.0 jfrog-platform global values.yaml file is 3.12.10-debian-11-r1, while the one used in the older versions of the chart is 3.11.10-debian-11-r5. This upgrade is affected by some missing featureFlags, ex. classic_mirrored_queue_version, that is not enabled in the older version, but required for the new version of rabbitmq.

After some debugging, we found out that there is a migration-hook.yaml file that has to create a job that will enable all the flags before the actual rabbitmq upgrade when the migration flag is set to true.

In our case, the job finishes with the following message: "Rabbitmq pod is not in running state. Ignoring feature flag migration for rabbitmq".

This is happening for 2 reasons:

  1. Namespace parameter is not added to the kubectl commands in the file, check this for example. I am not sure why -n {{ .Release.Namespace }} is not part of the commands because without it they won't give any results back and the migration job will terminate unsuccessfully.

  2. The condition below works when there is ONLY one rabbitmq instance because it evaluates the status of only one pod. In our case, we have two/three rabbitmq replicas, in staging/prod accordingly, and the command returns back the status of all of them, which means that the condition will be skipped and the job will again finish unsuccessfully:

$ kubectl -n jfrog-platform get pods -l "app.kubernetes.io/name=rabbitmq" -o jsonpath='{..status.conditions[?(@.type=="Ready")].status}'
True True

What you expected to happen:

This time we had to enable the flags manually in order to upgrade the platform, but we're expecting this to be done by the job.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

oumkale commented 5 months ago

Hi @dipop1,

Thank you for reporting this issue, It will be fixed in upcoming releases. We will update you

dipop1 commented 5 months ago

Thanks @oumkale,

Looking forward to hearing from you.