jhalterman / lyra

High availability RabbitMQ client
Apache License 2.0
262 stars 74 forks source link

Connection recovery fails and gives up even though recovery policy is recoverAlways() #91

Open amertahir opened 1 year ago

amertahir commented 1 year ago

After several attempts at recovery, the recovery fails on java.net.ConnectException:

[lyra-recovery-1] ERROR n.j.lyra.internal.ConnectionHandler - Failed to recover connection cxn-3
java.net.ConnectException: Connection refused (Connection refused)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
    at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
    at java.base/java.net.Socket.connect(Unknown Source)
    at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:60)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1137)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1027)
    at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:243)
    at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:236)
    at net.jodah.lyra.internal.RetryableResource.callWithRetries(RetryableResource.java:51)
    at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:236)
    at net.jodah.lyra.internal.ConnectionHandler.recoverConnection(ConnectionHandler.java:273)
    at net.jodah.lyra.internal.ConnectionHandler.access$100(ConnectionHandler.java:39)
    at net.jodah.lyra.internal.ConnectionHandler$ConnectionShutdownListener$1.run(ConnectionHandler.java:95)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

The connection retry and recovery policies are set to always retry and recover:

Config lyraConf = new Config()
    .withRecoveryPolicy(RecoveryPolicies.recoverAlways().withBackoff(Duration.seconds(1), Duration.seconds(20)))
    .withRetryPolicy(RetryPolicies.retryAlways().withBackoff(Duration.seconds(1), Duration.seconds(20)));

ConnectionOptions options = new ConnectionOptions(factory);
myClass.setConnection(Connections.create(options, lyraConf));
acogoluegnes commented 1 year ago

We don't have enough information to help. Please provide a standalone project and steps to reproduce the issue.

Note Lyra is no longer maintained, you should use the RabbitMQ Java client, which provide automatic connection and topology recovery.