search

arobson / rabbot

Deprecated: Please see https://github.com/Foo-Foo-MQ/foo-foo-mq
MIT License
276 stars 129 forks source link

Configure call across containers in a docker stack not establishing provided topology, only default response #108

Closed rhyslbw closed 6 years ago

rhyslbw commented 7 years ago

See reproduction here: https://github.com/rhyslbw/rabbot-configure-docker

Context Two services in a docker-compose file, a rabbot client using the configure api to connect to a rabbitmq:3-management broker in the stack via the default network.

Problem When bringing up the stack via docker-compose up only the default response topology is created after the connection is established. When the client service is run independently on my dev machine with ports mapped into the broker, it connects and configures the defined topology as expected.

Notes

➜  rabbot-configure-docker git:(master) docker-compose up            
Starting rabbotconfiguredocker_broker_1 ... 
Starting rabbotconfiguredocker_broker_1 ... done
Starting rabbotconfiguredocker_client_1 ... 
Starting rabbotconfiguredocker_client_1 ... done
Attaching to rabbotconfiguredocker_broker_1, rabbotconfiguredocker_client_1
client_1  | 2017-09-30T03:53:23.191Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.223Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.224Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.226Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:23.233Z [rabbot.topology] Failed to create reply queue for connection name 'default Error: Failed to create queue '10c696ec7ae1.node.1.response.queue' on connection 'default' with 'No endpoints could be reached'
client_1  |     at Topology.<anonymous> (/usr/src/app/node_modules/rabbot/src/topology.js:196:10)
client_1  |     at onConnectionFailed (/usr/src/app/node_modules/rabbot/src/topology.js:208:13)
client_1  |     at /usr/src/app/node_modules/rabbot/src/topology.js:214:6
client_1  |     at Object.invokeSubscriber (/usr/src/app/node_modules/monologue.js/lib/monologue.js:181:19)
client_1  |     at invoker (/usr/src/app/node_modules/monologue.js/lib/monologue.js:430:11)
client_1  |     at /usr/src/app/node_modules/monologue.js/lib/monologue.js:324:5
client_1  |     at arrayEach (/usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:1289:13)
client_1  |     at Function.<anonymous> (/usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:3345:13)
client_1  |     at /usr/src/app/node_modules/monologue.js/lib/monologue.js:436:7
client_1  |     at /usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:3073:15
client_1  |     at baseForOwn (/usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:2046:14)
client_1  |     at /usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:3043:18
client_1  |     at Function.<anonymous> (/usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:3346:13)
client_1  |     at fsm.emit (/usr/src/app/node_modules/monologue.js/lib/monologue.js:435:6)
client_1  |     at fsm.failed (/usr/src/app/node_modules/rabbot/src/connectionFsm.js:346:11)
client_1  |     at fsm.handle (/usr/src/app/node_modules/machina/lib/machina.js:613:25)
client_1  | No endpoints could be reached
client_1  | Done
client_1  | 2017-09-30T03:53:23.235Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.237Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.237Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.238Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:23.339Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.341Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.342Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.342Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:23.543Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.546Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.546Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.547Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:23.849Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.850Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.851Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.851Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:24 ===
broker_1  | Starting RabbitMQ 3.6.12 on Erlang 19.2.1
broker_1  | Copyright (C) 2007-2017 Pivotal Software, Inc.
broker_1  | Licensed under the MPL.  See http://www.rabbitmq.com/
broker_1  | 
broker_1  |               RabbitMQ 3.6.12. Copyright (C) 2007-2017 Pivotal Software, Inc.
broker_1  |   ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
broker_1  |   ##  ##
broker_1  |   ##########  Logs: tty
broker_1  |   ######  ##        tty
broker_1  |   ##########
broker_1  |               Starting broker...
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:24 ===
broker_1  | node           : rabbit@8ebcc31dee2b
broker_1  | home dir       : /var/lib/rabbitmq
broker_1  | config file(s) : /etc/rabbitmq/rabbitmq.config
broker_1  | cookie hash    : odpKFWjzmQ9v+NVANocB3w==
broker_1  | log            : tty
broker_1  | sasl log       : tty
broker_1  | database dir   : /var/lib/rabbitmq/mnesia/rabbit@8ebcc31dee2b
client_1  | 2017-09-30T03:53:24.253Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:24.255Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:24.255Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:24.255Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:24.761Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:24.762Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:24.762Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:24.763Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:25.367Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:25.369Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:25.369Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:25.369Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | Memory high watermark set to 799 MiB (838351257 bytes) of 1998 MiB (2095878144 bytes) total
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | Enabling free disk space monitoring
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | Disk free limit set to 50MB
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | Limiting to approx 1048476 file handles (943626 sockets)
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | FHC read buffering:  OFF
broker_1  | FHC write buffering: ON
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Waiting for Mnesia tables for 30000 ms, 9 retries left
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Waiting for Mnesia tables for 30000 ms, 9 retries left
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Priority queues enabled, real BQ is rabbit_variable_queue
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Starting rabbit_node_monitor
client_1  | 2017-09-30T03:53:26.070Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:26.073Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:26.073Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:26.073Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Management plugin: using rates mode 'basic'
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | msg_store_transient: using rabbit_msg_store_ets_index to provide index
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | msg_store_persistent: using rabbit_msg_store_ets_index to provide index
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | started TCP Listener on [::]:5672
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Management plugin started. Port: 15672
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Statistics database started.
broker_1  |  completed with 6 plugins.
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Server startup complete; 6 plugins started.
broker_1  |  * rabbitmq_management
broker_1  |  * rabbitmq_web_dispatch
broker_1  |  * rabbitmq_management_agent
broker_1  |  * amqp_client
broker_1  |  * cowboy
broker_1  |  * cowlib
client_1  | 2017-09-30T03:53:26.876Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | accepting AMQP connection <0.490.0> (172.20.0.3:54718 -> 172.20.0.2:5672)
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | connection <0.490.0> (172.20.0.3:54718 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
client_1  | 2017-09-30T03:53:26.896Z [rabbot.connection] Connected to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | connected
client_1  | 2017-09-30T03:53:26.910Z [rabbot.topology] Declaring queue '10c696ec7ae1.node.1.response.queue' on connection 'default' with the options: {"autoDelete":true,"subscribe":true,"uniqueName":"10c696ec7ae1.node.1.response.queue"}
client_1  | 2017-09-30T03:53:26.921Z [rabbot.topology] Declaring undefined exchange '' on connection 'default' with the options: {"passive":true}
client_1  | 2017-09-30T03:53:26.923Z [rabbot.queue] Starting subscription to queue '10c696ec7ae1.node.1.response.queue' on 'default'
client_1  | 2017-09-30T03:53:26.925Z [rabbot.queue] Subscription to (tracked) queue 10c696ec7ae1.node.1.response.queue - default started with consumer tag 10c696ec7ae1.node.1.response.queue
edorsey commented 6 years ago

I ran into this same problem as well. It appears that this module makes an assumption:

RabbitMQ will be running when rabbot tries to connect.

This is not the case with docker-compose.

To fix this, I did two things, after getting the connected event from rabbot. It is a bit hacky, but it appears to work:

let rabbit = require("rabbot")

let connectionConfig = {
  //connection details
};

rabbit.addConnection(connectionConfig) //Start by adding the connection only

rabbit.on("connected", function(a, b) {
  //Once we've connected, replace the connection promise with one that automatically resolves
  //Prior to this change, this is the original promise that failed. Publishing will always fail because of this.
  rabbit.connections[connectionConfig.name].promise = new Promise(function (resolve, reject) {
    resolve()
  })

  //Now reconfigure rabbit because if it fails its initial connection, the configuration doesn't ever get setup
  rabbit.configure({
    connection: connectionConfig,
    exchanges: [],
    queues: [],
    bindings:[]
  })
})

Not 100% that using the built-in Promise will cause any issues with the promises used in this project.

P.S. I looked at trying to fix this inside the module and submitting a PR, but I think it will require someone that better understands the internals to address. It might require a bit of re-working the internals from the bit of time I spent nailing down what was going on.

arobson commented 6 years ago

You can get rabbot to keep retrying by catching the failure and telling it to just start over. You can also tell it to increase the number of times it will attempt to connect to a broker before considering the broker "failed".

See the retryLimit and failAfter settings here to change failure semantics.

See this on telling it to reset and try again after reaching failure.

antmarot commented 6 years ago

I had the same issue.

It is true that more attempts to connect will be made but the promise returned by configure is rejecting on the first failed attempt.

I ended up doing this

const rabbot = require('rabbot');
const Promise = require('bluebird');

async function connect() {
  await rabbot.configure({
    connection: {
      /* connection details */
      retryLimit: 0, // no implicit retries
    },
    /* other options */
  });

  // Update internal promise - see @edorsey code
  const connectionName = 'default';
  rabbot.connections[connectionName].promise = Promise.resolve();
}

async function connectWithRetry(attempts = 1) {
  try {
    await connect();
  } catch (err) {
    if (attempts < 5) {
      console.log('Connection to RabbitMQ failed. Retrying in 5s...');
      await Promise.delay(5000);
      return connectWithRetry(attempts + 1);
    }
    throw err;
  }
}
try {
  await connectWithRetry();
} catch (err) {
  console.error(err);
  process.exit(1);
}
genemyers commented 5 years ago

@antmarot When I tried your code I received a UnhandledPromiseRejectionWarning when the connection is unreachable and the code exits. There's not a catch block on the configure promise, but I've had no luck adding one and making this work. Any insight would be appreciated.

antmarot commented 5 years ago

~@genemyers I added a snippet calling the connection logic. Hope this helps.~

@genemyers I ran into the same issue. I had to use the fix proposed by @edorsey. The code in my previous response has been adapted.

genemyers commented 5 years ago

@antmarot would you be so kind as to provide a single module that demonstrates your solution? I am getting an error that Promise.delay isn't a function.

antmarot commented 5 years ago

@genemyers I now included the module dependencies. Promise.delay is provided by bluebird.

genemyers commented 5 years ago

@antmarot I should have caught that, thanks, working now.