abreits / amqp-ts

AmqpSimple, a RabbitMQ tested Amqp library written in and for Typescript
MIT License
131 stars 42 forks source link

IllegalOperationError: Channel closed #49

Open sanzog03 opened 4 years ago

sanzog03 commented 4 years ago

I don't exactly know how but very often i see the tcp connection to the rabbitmq server keeps incresing gradually. The increasing connection is quite problematic as it floods the rabbitmq server where the live connection gets blocked. When the connection increases i can see the following logs porduced by amqp-ts in client application:

Unhandled rejection IllegalOperationError: Channel closed
     at Channel.<anonymous> (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/channel.js:149:11)
     at Channel.C._rpc (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/channel.js:131:8)
     at Channel.rpc (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/callback_model.js:73:8)
     at Channel.bindQueue (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/callback_model.js:121:17)
     at /tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqp-ts/lib/amqp-ts.js:1088:36
     at tryCatcher (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/bluebird/js/release/util.js:16:23)
     at Promise._settlePromiseFromHandler (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/bluebird/js/release/promise.js:547:31)
     at Promise._settlePromise (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/bluebird/js/release/promise.js:604:18)
     at Promise._settlePromiseCtx (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/bluebird/js/release/promise.js:641:10)
     at _drainQueueStep (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/bluebird/js/release/async.js:97:12)
     at _drainQueue (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/bluebird/js/release/async.js:86:9)
     at Async._drainQueues (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/bluebird/js/release/async.js:102:5)
     at Immediate.Async.drainQueues [as _onImmediate] (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/bluebird/js/release/async.js:15:14)
     at runCallback (timers.js:694:18)
     at tryOnImmediate (timers.js:665:5)
 Unhandled rejection Error: Channel ended, no reply will be forthcoming
     at rej (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/channel.js:190:7)
     at Channel.C._rejectPending (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/channel.js:192:28)
     at Channel.C.toClosed (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/channel.js:160:8)
     at Connection.C._closeChannels (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/connection.js:392:18)
     at Connection.C.toClosed (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/connection.js:399:8)
     at Connection.C.onSocketError (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/connection.js:353:10)
     at Connection.emit (events.js:182:13)
     at Connection.EventEmitter.emit (domain.js:442:20)
     at Socket.go (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/connection.js:479:12)
     at Socket.emit (events.js:182:13)
     at Socket.EventEmitter.emit (domain.js:442:20)
     at emitReadable_ (_stream_readable.js:534:12)
     at process._tickCallback (internal/process/next_tick.js:63:19)
 Unhandled rejection Error: Operation failed: QueueDeclare; 405 (RESOURCE-LOCKED) with message "RESOURCE_LOCKED - cannot obtain exclusive access to locked queue '54:b2:03:0b:12:d1_ota' in vhost '/'. It could be originally declared on another connection or the exclusive property value does not match that of the original declaration."
     at reply (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/channel.js:127:17)
     at Channel.C.accept (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/channel.js:401:7)
     at Connection.mainAccept [as accept] (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/connection.js:63:33)
     at Socket.go (/tmp/.mount_mps-ui3OFi3Q/resources/app.asar/node_modules/amqplib/lib/connection.js:476:48)

i.e. there seems to appear IllegalOperationError: Channel closed along with QueueDeclare; 405 (RESOURCE-LOCKED) unhandled rejection errors. In addition to that there is no exact steps to recreate the issue as of now, but this issue is arising quite often.

jelgersma commented 4 years ago

Having the same issue. We haven't found a solution yet. If we do, we will share it here. It seems that 5 services (where it was creating 5 connections before), it now creates above 100's of connections. Had any luck solving this yourself?

sanzog03 commented 4 years ago

I found some problem with the reconnection strategy that this library uses. i.e. when a client (using this library) faces a network connection issue, the auto reconnection startegy feature kicks in. A connection with explicit consumer tag couldn't even reconnect to the amqp server. I couldn't exactly replicate - with steps - the increasing amqp connections, but i saw the problem occuring only after network instability in the client devices. Also to mention, when the amqp server restarted after not being able to handle too much connections, the reconnection strategy could not make client connect to the restarted amqp server. In the mean time i came across amqplib library and becase of this problem i changed the whole amqp library of the production application i was working on with amqplib and handled the reconnection startegy by myself. So the library change (amqplib) and custom reconnection strategy has fixed the issue on my application.