unexpected_trailing_sections,{{'v1_0.delivery_annotations',

Hi,

I'm running a rabbitmq kubernetes cluster and would like to know where did i missed. this is the error.

<0.1193.0> exit with reason reached_max_restart_intensity in context shutdown
2020-09-27 19:01:29.332 [error] <0.1187.0> closing AMQP connection <0.1187.0> (10.1.7.1:60751 -> 10.1.7.199:5672):
{handshake_error,running,<0.1193.0>,{{symbol,<<"amqp:internal-error">>},"Session error: ~p~n~p~n",[{unexpected_trailing_sections,{{'v1_0.delivery_annotations',[]},<<0,83,114,193,1,0,0,83,115,208,0,0,0,18,0,0,0,3,64,64,161,10,116,101,115,116,45,113,117,101,117,101,0,83,116,193,1,0,0,83,119,161,5,104,101,108,108,111,0,83,120,193,1,0>>}},[{rabbit_amqp1_0_message,assemble,4,[{file,"src/rabbit_amqp1_0_message.erl"},{line,104}]},{rabbit_amqp1_0_message,assemble,1,[{file,"src/rabbit_amqp1_0_message.erl"},{line,21}]},{rabbit_amqp1_0_incoming_link,transfer,4,[{file,"src/rabbit_amqp1_0_incoming_link.erl"},{line,138}]},{rabbit_amqp1_0_session_process,handle_control,2,[{file,"src/rabbit_amqp1_0_session_process.erl"},{line,253}]},{rabbit_amqp1_0_session_process,handle_cast,2,[{file,"src/rabbit_amqp1_0_session_process.erl"},{line,173}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]]}}

and here is my kubernetes config.

apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
data:
  enabled_plugins: |
    [rabbitmq_federation,rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_amqp1_0].
  rabbitmq.conf: |
    loopback_users.guest = false
    listeners.tcp.default = 5672
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    cluster_formation.node_cleanup.only_log_warning = true
    ##cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
    ##cluster_formation.classic_config.nodes.1 = rabbit@rabbitmq-0.rabbitmq.rabbits.svc.cluster.local
    ##cluster_formation.classic_config.nodes.2 = rabbit@rabbitmq-1.rabbitmq.rabbits.svc.cluster.local
    ##cluster_formation.classic_config.nodes.3 = rabbit@rabbitmq-2.rabbitmq.rabbits.svc.cluster.local
    ##rabbitmq_amqp1_0
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq
rules:
- apiGroups: 
    - ""
  resources: 
    - endpoints
  verbs: 
    - get
    - list
    - watch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq
  namespace: rabbits
subjects:
- kind: ServiceAccount
  name: rabbitmq
  namespace: rabbits
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rabbitmq
---
apiVersion: v1
kind: Secret
metadata:
  name: rabbit-secret
type: Opaque
data:
  # echo -n "cookie-value" | base64
  RABBITMQ_ERLANG_COOKIE: V0lXVkhDRFRDSVVBV0FOTE1RQVc=
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
spec:
  serviceName: rabbitmq
  replicas: 2
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      serviceAccountName: rabbitmq
      initContainers:
      - name: config
        image: busybox
        command: ['/bin/sh', '-c', 'cp /tmp/config/rabbitmq.conf /config/rabbitmq.conf && ls -l /config/ && cp /tmp/config/enabled_plugins /etc/rabbitmq/enabled_plugins']
        volumeMounts:
        - name: config
          mountPath: /tmp/config/
          readOnly: false
        - name: config-file
          mountPath: /config/
        - name: plugins-file
          mountPath: /etc/rabbitmq/
      containers:
      - name: rabbitmq
        image: rabbitmq:3-management-alpine
        ports:
        - containerPort: 4369
          name: discovery
        - containerPort: 5672
          name: amqp
        - containerPort: 15672
          name: port-mgmt
        env:
        - name: RABBIT_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: RABBIT_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_NODENAME
          value: rabbit@$(RABBIT_POD_NAME).rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_USE_LONGNAME 
          value: "true"
        - name: RABBITMQ_CONFIG_FILE
          value: "/config/rabbitmq"
        - name: RABBITMQ_ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: rabbit-secret
              key: RABBITMQ_ERLANG_COOKIE
        - name: K8S_HOSTNAME_SUFFIX
          value: .rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_DEFAULT_USER
          value: rabbit-user
        - name: RABBITMQ_DEFAULT_PASS
          value: rabbit-pass        
        volumeMounts:
        - name: data
          mountPath: /var/lib/rabbitmq
          readOnly: false
        - name: config-file
          mountPath: /config/
        - name: plugins-file
          mountPath: /etc/rabbitmq/
      volumes:
      - name: config-file
        emptyDir: {}
      - name: plugins-file
        emptyDir: {}
      - name: config
        configMap:
          name: rabbitmq-config
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "microk8s-hostpath"
      resources:
        requests:
          storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
spec:
  clusterIP: None
  ports:
  - port: 4369
    targetPort: 4369
    name: discovery
  - port: 5672
    targetPort: 5672
    name: amqp
  selector:
    app: rabbitmq
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-svc
  labels:
    app: rabbitmq
spec:
  type: NodePort 
  ports:
  - port: 15672
    name: port-mgmt
    nodePort: 30006
    protocol: TCP
  - port: 5672
    name: port-actual
    nodePort: 30007
    targetPort: 5672
    protocol: TCP
  selector:
    app: rabbitmq

I also tried using the latest aside from this image i'm using but simillar output.

I'm getting this error log whenever i'm listening or consuming the message in my application which is quarkus. Also, i added a plugin rabbitmq_amqp1_0 as the library i'm using currently supports that.

Not really sure which is causing this error so i decided to raise this here to have a better understanding.

Thanks

rabbitmq / rabbitmq-management

unexpected_trailing_sections,{{'v1_0.delivery_annotations', #851