taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
https://bullmq.io
MIT License
6.16k stars 405 forks source link

Reuse connection with lazyConnect on worker #281

Open nmrgt opened 4 years ago

nmrgt commented 4 years ago

If I use a shared redis connection with lazy connect option in a worker, like this (pseudo code)

const client = new Redis(connectionString, { lazyConnect: true });
await client.connect();
const worker = new Worker('myworker', () => {}, { connection: client });

Worker dont process any job exept those already in queue when starting. blockingConnection.client promise is never resolved which block every new job processing.

In Worker constructor :

    this.blockingConnection = new RedisConnection(
      isRedisInstance(opts.connection)
        ? (<Redis>opts.connection).duplicate()
        : opts.connection,
    );

I think that the bug come from redis duplicating (clone redis object with options - like lazyconnect) but don't initialize connection. Maybe we could fix this by :

The quick fix for me was to remove lazyconnect :)

Thanks for your great work !

manast commented 4 years ago

Could be a bug in ioredis when duplicating a connection that has the lazyConnect: true option set. Best would be to just warn about it I think.

toverux commented 1 year ago

I am also using lazyConnect and I cannot observe this behavior, it's probably been fixed upstream?


I stumbled on this issue as I was trying to understand why BullMQ duplicates connections. Probably a design choice, but it might cause certain problems.

The use case is repeated creation and destruction of QueueEvents objects. My app leaked connections as I passed customized, ready-to-use client instances to Bull, without realising that Bull duplicated connections. I closed my own ioredis connections instead of using Bull APIs, so 1) I leaked a ton of connections (fixed by using BullMQ's close() methods) and 2) I connected two times to the server (fixed with lazyConnect).

I would prefer that Bull did not duplicated connections implictly, this goes against the principle of least astonishment, and I prefer to manage my ioredis connections. At least, an option to control this would be nice.