HTTPImporter not quitting after AMQP failure

[TimVosch]

2024/01/08 19:43:07 AMQPConnection connect failed: dial tcp 10.0.82.73:5672: connect: connection refused
2024/01/08 19:43:07 AMQPConnection retry in 24 seconds...
2024/01/08 19:43:31 AMQPConnection (re)connecting...
2024/01/08 19:43:31 AMQPConnection connect failed: dial tcp 10.0.82.73:5672: connect: connection refused
2024/01/08 19:43:31 AMQPConnection retry in 27 seconds...
2024/01/08 19:43:58 AMQPConnection (re)connecting...
2024/01/08 19:43:58 AMQPConnection connect failed: dial tcp 10.0.82.73:5672: connect: connection refused
2024/01/08 19:43:58 AMQPConnection retry in 30 seconds...
2024/01/08 19:44:28 AMQPConnection (re)connecting...
2024/01/08 19:44:28 AMQPConnection connect failed: dial tcp 10.0.82.73:5672: connect: connection refused
2024/01/08 19:44:28 AMQPConnection maximum retries of 11 reached, quitting...
2024/01/08 19:44:28 AMQPConnection stopping
2024/01/08 19:44:28 AMQPConnection stopped
2024/01/08 19:44:28 AMQPPublisher lost connection

[JEFFTheDev]

AMQP Connection stops retrying after 11 retries, however the service keeps running. There is no indication for our infrastructure that the service is unhealthy and needs to be restarted or looked into.

Reproduce:

Happens to all services that use AMQP however example was made using HTTP importer

• Start service
• Make AMQP unreachable
• Await 11 retries (max)

Possible solutions:

• Pass cancelToken for service to AMQP connection and at max retries, stop the service by filling the cancelToken
• Increase max retries and have the amount of retries influence the value of a health-check endpoint which our infrastructure could use to indicate anything wrong

[JEFFTheDev]

Update:

Opted for health endpoint. This solution seems more maintainable for when we add more connections to other (3rd party) services

PR: https://github.com/sensorbucket/SensorBucket/pull/73