[bitnami/rabbitmq] Missing upgrade guide to v12.0.0 (RabbitMQ 3.11->3.12)

jerr0328 commented 1 year ago

Name and Version

bitnami/rabbitmq 12.0.x

What architecture are you using?

amd64

What steps will reproduce the bug?

Install bitnami/rabbitmq chart with persistence using chart version 11.x
Attempt to upgrade to chart version 12.0.x
Upgrade will fail due to missing feature flags

Are you using any custom parameters or values?

clustering:
  enabled: false
metrics:
  enabled: false
# Remove clustering-related plugins by specifying only needed plugin
plugins: "rabbitmq_management"
# Remove LDAP extra plugin (set in defaults)
extraPlugins: ""
# Avoid PRECONDITION_FAILED: https://www.rabbitmq.com/consumers.html#acknowledgement-timeout
extraConfiguration: |
  consumer_timeout = 7200000
resources:
  limits:
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 512Mi
# If this is not enabled, RabbitMQ thinks it has way more memory than what is allowed by K8s
memoryHighWatermark:
  enabled: true
  type: "relative"
  value: 0.8

What is the expected behavior?

I would usually expect that an upgrade could be done without needing to manually specify anything. If there are new feature flags, the chart should provide those as default enabled to allow the upgrade to happen. At the very least, upgrade instructions in the README should point out that you can't upgrade from 11 to 12 without taking manual actions

What do you see instead?

rabbitmq 08:58:14.35 
rabbitmq 08:58:14.35 Welcome to the Bitnami rabbitmq container
rabbitmq 08:58:14.35 Subscribe to project updates by watching https://github.com/bitnami/containers
rabbitmq 08:58:14.36 Submit issues and feature requests at https://github.com/bitnami/containers/issues
rabbitmq 08:58:14.36 
rabbitmq 08:58:14.36 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq 08:58:14.37 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq 08:58:14.39 INFO  ==> Initializing RabbitMQ...
rabbitmq 08:58:14.42 INFO  ==> Persisted data detected. Restoring...
rabbitmq 08:58:14.42 INFO  ==> No custom scripts in /docker-entrypoint-initdb.d
rabbitmq 08:58:14.42 INFO  ==> ** RabbitMQ setup finished! **

rabbitmq 08:58:14.44 INFO  ==> ** Starting RabbitMQ **
2023-06-19 08:58:15.964181+00:00 [error] <0.235.0> Feature flags: `classic_mirrored_queue_version`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
2023-06-19 08:58:15.970811+00:00 [error] <0.235.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
2023-06-19 08:58:15.970811+00:00 [error] <0.235.0>                                               classic_mirrored_queue_version}

BOOT FAILED
2023-06-19 08:58:15.976435+00:00 [error] <0.235.0> 
2023-06-19 08:58:15.976435+00:00 [error] <0.235.0> BOOT FAILED
===========
Error during startup: {error,failed_to_initialize_feature_flags_registry}

2023-06-19 08:58:15.976435+00:00 [error] <0.235.0> ===========
2023-06-19 08:58:15.976435+00:00 [error] <0.235.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
2023-06-19 08:58:15.976435+00:00 [error] <0.235.0> 
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>   crasher:
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     initial call: application_master:init/4
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     pid: <0.234.0>
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     registered_name: []
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     exception exit: {failed_to_initialize_feature_flags_registry,
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>                         {rabbit,start,[normal,[]]}}
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>       in function  application_master:init/4 (application_master.erl, line 142)
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     ancestors: [<0.233.0>]
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     message_queue_len: 1
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     messages: [{'EXIT',<0.235.0>,normal}]
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     links: [<0.233.0>,<0.44.0>]
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     dictionary: []
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     trap_exit: true
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     status: running
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     heap_size: 233
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     stack_size: 28
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>     reductions: 168
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0>   neighbours:
2023-06-19 08:58:16.977133+00:00 [error] <0.234.0> 
2023-06-19 08:58:16.981119+00:00 [notice] <0.44.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...

Additional information

No response

aoterolorenzo commented 1 year ago

Hi @jerr0328

Seems it was an automatic upgrade so no guided steps are currently defined. I will open a task for the team to dive in the upgrade details and write those steps as soon as our workload allows us.

EvgeniGordeev commented 1 year ago

rabbitmqctl enable_feature_flag all will do it. Tested with the upgrade from 3.11.20 to 3.12.2.

When in a cluster running the command on the main node is enough:

kubectl exec -it rabbitmq-0 -- rabbitmqctl enable_feature_flag all

emily-zall commented 6 months ago

I ran into this too, is there a way to do this without running a manual command? I tried adding featureFlags: "classic_mirrored_queue_version" to my values but I still got this error

also the manual command isn't working for me because my pod is crashing before I can shell into it

emily-zall commented 6 months ago

featureFlags: "classic_mirrored_queue_version" should be added before the upgrade. We ended up deleting the pvc (unknown what data is lost but this was okay on our dev env) to spin up cleanly since we had not been aware of this.

tadasgedgaudas commented 5 months ago

Just enable all feature flags on your current version and then do upgrade, works perfectly

N4SoftwareNinja commented 4 months ago

I think adding a paragraph like this should do the trick, because in the RabbitMQ instructions it is mentioned that all feature flags should be enabled.

To 12.0.0

This major version changes the default RabbitMQ image from 3.11.x to 3.12.x. Follow the official instructions to upgrade from 3.11 to 3.12.

carrodher commented 4 months ago

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

bitnami / charts