Error: Sending custom commands in pipeline is not supported in Cluster mode when use Cluster Mode

[zetflo]

When i use bull in redis cluster mode.

Configuration options

const opts = {
    createClient: function(type){
       return new redis.Cluster([
            ...
        ]);
    },
    prefix: '{}'
};

Create job it's worked but when i process job with queue.process, I encounter the following error

Error: Sending custom commands in pipeline is not supported in Cluster mode.
    at Pipeline.exec (\node_modules\ioredis\lib\pipeline.js:249:19)
    at Pipeline.pipeline.exec (\node_modules\ioredis\lib\transaction.js:34:26)
    at \node_modules\bull\lib\job.js:210:18

Apparently the pipelines are not compatible with cluster mode of redis.

Can you confirm this problem? How to use bull with a cluster redis without encountering this problem?

[manast]

ok, seems like it is a limitation of ioredis at least. I did not now about it: https://github.com/luin/ioredis/issues/536 This requires some rewrite in order to make it work in cluster mode unfortunately.

[ajwootto]

@manast we are starting to run into this. Do you know what features specifically require pipelines? Is there some subset of features we can still use without them?

[btzsoft]

@manast I'm running in the same issue when calling job.moveToFailed(err). it's possible to do something else here? it's strange for me, because I use job.moveToCompleted() without any problems. thx!

[manast]

@btzsoft that is because ioredis does not support multi commands using custom lua scripts, and only moveToFailed currently uses that. In order to solve this issue the method should be rewritten to not use multi, which is a bit to work to do...

[btzsoft]

@manast oh no, do you have some plans in future to change this? if no can you give pls some hints where to take a look, maybe I will make a fork and will changes this, bcs it's really a big blocker for us. thank you for your time.

[manast]

you can check the implementation of moveToFailed, it is quite inconvenient to not being able to use multi in this case. The other alternative is to convince the authors of ioredis to fix this issue: https://github.com/luin/ioredis/issues/536

[btzsoft]

@manast just to better understand, why moveToFailed use multi command? and why doesn't work as moveToFinished. thx!

[manast]

@btzsoft I think you could understand this better if you check the source code, basically it is because moveToFailed has much more functionality such as retries, updating attempts, backoffs, etc.

[joebowbeer]

@manast Does this mean bull is incompatible with a Redis cluster-mode-enabled cluster until ioredis is enhanced? Is there no workaround?

In the meantime, can you update the Patterns doc, which provides a (broken?) example of how to use Redis Cluster?

https://github.com/OptimalBits/bull/blob/develop/PATTERNS.md#redis-cluster

[akhil-karat]

@manast - I am facing same problem. Works with local redis instance. But in our int and prod we use Redis cluster. When a job is added to queue process fails with error 'Sending custom commands in pipeline is not supported in Cluster mode'. Then job stalls. Is there a work around for this issue? It is really urgent for us. Kindly help.

[joebowbeer]

@manast Can you confirm that this is not an issue in BullMQ?

[manast]

@joebowbeer it is the same problem unfortunately. I was expecting ioredis to be fixed but for some reason they did not prioritize it. The alternative is to convert "moveToFailed" method in bull/bullMQ to one lua script, as you can see it is quite a bit of job, but it seems like it is the only way to solve this issue once and for all... will try to prioritize it.

[manast]

@akhil-karat

@manast - I am facing same problem. Works with local redis instance. But in our int and prod we use Redis cluster. When a job is added to queue process fails with error 'Sending custom commands in pipeline is not supported in Cluster mode'. Then job stalls. Is there a work around for this issue? It is really urgent for us. Kindly help.

Are you sure this happens when you add the job and not when the job fails?

[joebowbeer]

@manast Thanks for prioritizing this! I repo'd the issue in BullMQ as well.

[Rua-Yuki]

Super sorry for the excessive mentions all! (Was doing a bit of cleanup and of course GH has appended to this discussion for every instance of these commits.)

That said, if anyone urgently needs this working, a patch can be found in the fpm-git/bull/bug/fix-#758-redis-cluster-breakage branch.

Insert the usual "it works for us, but no warranties, etc." spiel here.

[joebowbeer]

@manast I think the ioredis issue is fixed in v4.19.0

[huxwfun]

@Rua-Yuki it appears addBulk uses pipeline as well

Job.createBulk = function(queue, jobs) {
  jobs = jobs.map(job => new Job(queue, job.name, job.data, job.opts));

  return queue
    .isReady()
    .then(() => {
      const multi = queue.client.multi();

[manast]

Unfortunately the fix the did on ioredis breaks Bulls unit tests for repeatable jobs, so we cannot use that fix until we discover the reason for the breakage.

[Rua-Yuki]

@Rua-Yuki it appears addBulk uses pipeline as well

Job.createBulk = function(queue, jobs) {
  jobs = jobs.map(job => new Job(queue, job.name, job.data, job.opts));

  return queue
    .isReady()
    .then(() => {
      const multi = queue.client.multi();

@huxwfun Many thanks for bringing this to my attention! Our use case only demanded a quick fix for the issue raised here with moveToFailed, though I can definitely see about adding support for pipeline-free addBulk as well (potentially sometime early next week).

Alternatively, https://github.com/OptimalBits/bull/pull/1933 can be used to get the benefits of ioredis 4.19.0+, which I believe should fix this sort of issue without requiring elimination of scripts from pipelines.

---

Unfortunately the fix the did on ioredis breaks Bulls unit tests for repeatable jobs, so we cannot use that fix until we discover the reason for the breakage.

I've just added https://github.com/OptimalBits/bull/pull/1933, which seems to have the repeatable job unit tests all happy and green with ioredis 4.19.0+.

Please don't hesitate to let me know if there are any further steps I should take, or anything I might do to help out. Thanks!

[gurbaj5124871]

Is this issue resolved by updating the bull and ioredis because it seemed to fix it for me, should this issue be closed and resolved?

[manast]

@gurbaj5124871 I am not 100% sure. Maybe somebody else from this thread could verify it before we close.