search

reportportal / service-api

Report portal. Main API Service
Apache License 2.0
44 stars 65 forks source link

[API][MQ] The connection to MQ is not recreated when receiving the error `#method<channel.close>` #1945

Closed shaburov closed 6 months ago

shaburov commented 6 months ago

Precondition

RabbitMQ cluster ![image](https://github.com/reportportal/service-api/assets/15269131/a08de1ed-7106-455b-859d-dd1430310f4b) ![image](https://github.com/reportportal/service-api/assets/15269131/ad83c911-7319-465d-a0f1-fb8729d62b64)

service-api -> 3 nodes version: 5.11.0

service-api MQ connection properties ``` 'rp.amqp.addresses': 'amqp://rabbitmq:password@reportportal[2z3|1z4|3z1].h.foo.bar:5672' 'rp.amqp.api-address': 'http://rabbitmq:password@reportportal[2z3|1z4|3z1].h.foo.bar:15672/api' 'rp.amqp.user': 'rabbitmq' 'rp.amqp.pass': 'password' 'rp.amqp.apiuser': 'rabbitmq' 'rp.amqp.apipass': 'password' 'rp.amqp.analyzer-vhost': 'analyzer' ```

When the service-api cluster starts, the first instance of three connects to MQ without errors. The remaining 2 instances are riddled with errors: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED

More info ``` 2024-03-13 09:32:53.627 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.7' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.627 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.2' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.628 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.0' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.628 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.6' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.628 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.5' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.628 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.4' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.628 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.1' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.631 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.9' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.634 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.3' in vhost '/' in exclusive use, class-id=60, method-id=20) 2024-03-13 09:32:53.634 INFO Shutdown Signal: channel error; protocol method: #method( reply-code=403, reply-text=ACCESS_REFUSED - queue 'reporting.8' in vhost '/' in exclusive use, class-id=60, method-id=20) ```

image

The error occurs because the connection to the queue is created with the listenerContainer.setExclusive(true) setting. Within one consumer group (with a static name) this is not required. Otherwise, there is no way to raise a cluster of several service-api instances that works with the RabbutMQ cluster.

Pink-Bumblebee commented 6 months ago

Please read https://reportportal.io/docs/installation-steps/ScalingUpReportPortalAPIService/.