Unable to publish after call to retry() when rabbit becomes "unreachable"

[brainsiq]

I am handling the 'unreachable' event and calling retry in an attempt to always have an active RabbitMQ connection. I normally wouldn't do this but shutting down the process isn't an option here.

When rabbot connects again I am unable to publish a message on the connection. The publish Promise is rejected with:

Publish error Error: Failed to create exchange 'test' on connection 'default' with 'No endpoints could be reached' at Topology. (/home/chrisi/code/playground/rabbit-retry/node_modules/rabbot/src/topology.js:189:12) at onConnectionFailed (/home/chrisi/code/playground/rabbit-retry/node_modules/rabbot/src/topology.js:201:16) at /home/chrisi/code/playground/rabbit-retry/node_modules/rabbot/src/topology.js:207:11 at SubscriptionDefinition.invokeSubscriber (/home/chrisi/code/playground/rabbit-retry/node_modules/monologue.js/lib/monologue.js:181:19) at invoker (/home/chrisi/code/playground/rabbit-retry/node_modules/monologue.js/lib/monologue.js:430:11) at arrayEach (/home/chrisi/code/playground/rabbit-retry/node_modules/lodash/index.js:1289:13) at Function. (/home/chrisi/code/playground/rabbit-retry/node_modules/lodash/index.js:3345:13) at fsm.emit (/home/chrisi/code/playground/rabbit-retry/node_modules/monologue.js/lib/monologue.js:439:6) at fsm.failed (/home/chrisi/code/playground/rabbit-retry/node_modules/rabbot/src/connectionFsm.js:345:16) at fsm.handle (/home/chrisi/code/playground/rabbit-retry/node_modules/machina/lib/machina.js:613:25) at fsm.Fsm.(anonymous function) [as handle] (/home/chrisi/code/playground/rabbit-retry/node_modules/machina/lib/machina.js:466:63) at fsm. (/home/chrisi/code/playground/rabbit-retry/node_modules/machina/lib/machina.js:702:17) at arrayEach (/home/chrisi/code/playground/rabbit-retry/node_modules/lodash/index.js:1289:13) at Function. (/home/chrisi/code/playground/rabbit-retry/node_modules/lodash/index.js:3345:13) at fsm.processQueue (/home/chrisi/code/playground/rabbit-retry/node_modules/machina/lib/machina.js:701:6) at fsm.Fsm.(anonymous function) [as processQueue] (/home/chrisi/code/playground/rabbit-retry/node_modules/machina/lib/machina.js:466:63)

Here is the code to reproduce:

const rabbot = require('rabbot');
let failed = false;

rabbot.on('unreachable',  () => {
    failed = true;
    console.error('RabbitMQ connection became unreachable. Forcing retry.');

    rabbot.retry()
        .catch(err => console.error('RabbitMQ connection retry failed'));
});

rabbot.on('failed', () => {
    console.error('RabbitMQ connection failed. Will automatically retry connection if limit not reached.');
});

rabbot.on('connected', () => {
    console.log('RabbitMQ connected.');

    if (failed) {
       // Try to publish a message after reconnecting
        rabbot.publish('test', { body: 'foo' })
            .then(() => console.log('Published'))
            .catch(err => console.error('Publish error', err));
    }
});

rabbot.configure({
    connection: {},
    exchanges: [
        { name: 'test', type: 'direct', durable: true, persistent: true }
    ]
});

I have been testing by running RabbitMQ, excuting the above code, then restarting the RabbitMQ service.

I see a lot of old closed issues relating to this error but none of them have helped resolve this. The only way I can get this to work is to call retry() again in the connected event before publishing, and even this only works if I delay it using setTimeout. I don't really want to rely on this as I don't know the significance of the delay.

Appreciate any help you can offer.

[brainsiq]

This also happens if you run the test script whilst RabbitMQ is not running, and then start RabbitMQ. When rabbot finally connects it is unable to publish and reports the same error.

[danielkcz]

Is there some way to catch that exception No endpoints could be reached? We are hosting on Heroku which is apparently more strict and without handling that exception it will crash the app and won't even bother to reboot it.

I tried to look through the code but it looks like it's happening very deeply and it's not exposed as a promise anywhere.