cannot find and enable oauth2 plugin

[h0jeZvgoxFepBQ2C]

Describe the bug

I created a cluster and want to enable oauth2 plugin afterwards.

To Reproduce

Create a cluster without oauth2 enabled at the beginning, then connect to a rabbitmq node and run following. It lists the oauth plugin, but it cannot enable it, since the plugin is not found?

rabbitmq-cluster-server-0:/etc/rabbitmq$ rabbitmq-plugins list
Listing plugins with pattern ".*" ...
WARNING - plugins currently enabled but missing: rabbit_auth_backend_oauth2

 Configured: E = explicitly enabled; e = implicitly enabled
 | Status: * = running on rabbit@rabbitmq-cluster-server-0.rabbitmq-cluster-nodes.rabbitmq-cluster
 |/
[  ] rabbitmq_amqp1_0                  3.10.13
[  ] rabbitmq_auth_backend_cache       3.10.13
[  ] rabbitmq_auth_backend_http        3.10.13
[  ] rabbitmq_auth_backend_ldap        3.10.13
[  ] rabbitmq_auth_backend_oauth2      3.10.13
[  ] rabbitmq_auth_mechanism_ssl       3.10.13
[  ] rabbitmq_consistent_hash_exchange 3.10.13
[  ] rabbitmq_event_exchange           3.10.13
[  ] rabbitmq_federation               3.10.13
[  ] rabbitmq_federation_management    3.10.13
[  ] rabbitmq_jms_topic_exchange       3.10.13
[E*] rabbitmq_management               3.10.13
[e*] rabbitmq_management_agent         3.10.13
[  ] rabbitmq_mqtt                     3.10.13
[  ] rabbitmq_peer_discovery_aws       3.10.13
[e*] rabbitmq_peer_discovery_common    3.10.13
[  ] rabbitmq_peer_discovery_consul    3.10.13
[  ] rabbitmq_peer_discovery_etcd      3.10.13
[E*] rabbitmq_peer_discovery_k8s       3.10.13
[E*] rabbitmq_prometheus               3.10.13
[  ] rabbitmq_random_exchange          3.10.13
[  ] rabbitmq_recent_history_exchange  3.10.13
[  ] rabbitmq_sharding                 3.10.13
[  ] rabbitmq_shovel                   3.10.13
[  ] rabbitmq_shovel_management        3.10.13
[e*] rabbitmq_stomp                    3.10.13
[  ] rabbitmq_stream                   3.10.13
[  ] rabbitmq_stream_management        3.10.13
[  ] rabbitmq_top                      3.10.13
[  ] rabbitmq_tracing                  3.10.13
[  ] rabbitmq_trust_store              3.10.13
[e*] rabbitmq_web_dispatch             3.10.13
[  ] rabbitmq_web_mqtt                 3.10.13
[  ] rabbitmq_web_mqtt_examples        3.10.13
[E*] rabbitmq_web_stomp                3.10.13
[  ] rabbitmq_web_stomp_examples       3.10.13

rabbitmq-cluster-server-0:/etc/rabbitmq$ rabbitmq-plugins enable rabbitmq_auth_backend_oauth2
Enabling plugins on node rabbit@rabbitmq-cluster-server-0.rabbitmq-cluster-nodes.rabbitmq-cluster:
rabbitmq_auth_backend_oauth2
Error:
{:plugins_not_found, [:rabbit_auth_backend_oauth2]}

This is my final yaml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-cluster
  namespace: rabbitmq-cluster
  annotations:
    rabbitmq.com/topology-allowed-namespaces: "*"
spec:
  replicas: 5
  persistence:
    storage: "1Gi"
  rabbitmq:
    additionalPlugins:
      - rabbitmq_web_stomp
      - rabbit_auth_backend_oauth2
    additionalConfig: |
      log.console = true
      log.console.level = debug

      auth_backends.1 = oauth2
      auth_backends.2 = internal

      cluster_partition_handling = pause_minority

      web_stomp.ws_frame = binary
      vm_memory_high_watermark.relative = 0.8
    #   management.path_prefix = /rabbit
    # 
    advancedConfig: |
      [
        {rabbit, [
          {auth_backends, [rabbit_auth_backend_oauth2, rabbit_auth_backend_internal]}
        ]},
        {rabbitmq_management, [
          {enable_uaa, false}
        ]},
        {rabbitmq_auth_backend_oauth2, [
          {resource_server_id, <<"rabbitmq-cluster">>},
          {resource_server_type, <<"rabbitmq">>},

          {key_config, [
            {default_key, <<"key">>},
            {signing_keys, #{
              <<"key">> => {pem, <<"-----BEGIN PUBLIC KEY-----
      key
      -----END PUBLIC KEY-----">>}
              }}
          ]}
        ]}
      ].

  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: rabbitmq-cluster
          topologyKey: kubernetes.io/hostname
  resources:
    requests:
      cpu: "1"
      memory: 3Gi
    limits:
      #cpu: 800m
      memory: 3Gi

Expected behavior It should find the plugin and enable it.

Version and environment information

• RabbitMQ: 3.10.13
• Erlang 25.2
• RabbitMQ Cluster Operator: 3.2.2

More context from the node

The plugin is existing in the file system?

rabbitmq-cluster-server-0:/$ rabbitmq-plugins directories -s

Plugin archives directory: /opt/bitnami/rabbitmq/plugins
Plugin expansion directory: /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-cluster-server-0.rabbitmq-cluster-nodes.rabbitmq-cluster-plugins-expand
Enabled plugins file: /operator/enabled_plugins

rabbitmq-cluster-server-0:/$ ls /opt/bitnami/rabbitmq/plugins | grep oauth
rabbitmq_auth_backend_oauth2-3.10.13

rabbitmq-cluster-server-0:/$ cat /operator/enabled_plugins
[rabbitmq_peer_discovery_k8s,rabbitmq_prometheus,rabbitmq_management,rabbitmq_web_stomp,rabbit_auth_backend_oauth2].

[h0jeZvgoxFepBQ2C]

Arrrr!

I found the issue: it seems that the old module name (before merge into rabbitmq core) was "rabbit_auth_backend_oauth2". now it is "rabbitmq_auth_backend_oauth2" (rabbitmq instead of rabbit).

Arrr that took long to find...

[mkuratczyk]

It has always been rabbitmq_auth_backend_oauth2. Anyway, glad you found the problem. :)