Loss of progress events

[bitliner]

• my queue has 3 workers
• each worker processes a task that can last few minutes (e.g. 1-3 minutes)

• each worker, while processing a task, sends a progress

  • about every 2/3 seconds
  • the payload of each progress event is few MB (~1.5MB)

The client that receives the progress does not receive all the events.

Somewhere the progress events are lost.

Any recommendation to better investigate?

[bitliner]

One thing that I've noted is that in Redis log I have the following error:

events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits.

related to https://redis.io/topics/clients#output-buffers-limits

I think Redis is receiving too much data and too fast (as set in the configuration), and Redis drops the connection with the redis client (kue).

Same thing happens when:

• connect to redis using redis-cli
• subscribe to the job events with the command subscribe q:events

==> after some seconds the connection drops.

Ways to solve it: scale the client, increase the pubsub buffer limit, e.g. 512 MB config set client-output-buffer-limit "pubsub 536870912 536870912 60"