[Bug]: Some keys appear again after queue obliterates

[shan-mx]

Version

v5.8.4

Platform

NodeJS

What happened?

Hello BullMQ team,

I've encountered an issue when using the queue.obliterate() method. In my case, despite the Redis keys associated with the queue are being deleted immediately, some of them will appear again after a period of delay.

How to reproduce.

import { Queue, Worker } from "bullmq";
import { exit } from "node:process";
import { sleep } from "radash";

const client = async () => {
  const queue = new Queue("my-queue");

  await queue.add("my-job", { foo: "bar" });

  const replyReceiver = new Worker(
    "my-queue-reply",
    async (job) => {
      console.log(job.data);
    },
    {
      connection: {
        host: "localhost",
        port: 6379,
      },
    },
  );

  await sleep(1000);

  const replyQueue = new Queue("my-queue-reply");

  await replyQueue.obliterate();

  await sleep(60_000);
};

const server = async () => {
  const worker = new Worker(
    "my-queue",
    async (job) => {
      console.log(job.data);
      const replyQueue = new Queue("my-queue-reply");
      await replyQueue.add("my-queue-reply", job.data);
    },
    {
      connection: {
        host: "localhost",
        port: 6379,
      },
    },
  );

  await sleep(60_000);
};

await Promise.all([client(), server()]);

exit();

The code's behavior: all keys related to reply queue will be deleted at first, after ~30 secs, the 'stalled-check' and 'meta' key will appear again.

Relevant log output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[roggervalf]

hi @shan-mx this is normal as maxLenEvents is backfilled with a default value when no value is found as soon as one of our scripts is executed. Same situation with stalled-check

[shan-mx]

@roggervalf thanks for the reply! I understand your words, which means a living queue usually need ~6 keys to describe its state. The maxLenEvents and stalled-check keys, and so on, have reason to exist. Another common fact is a deleted queue will not need these keys to describe it anymore, and deleting them is the job of obliterate().

However, The strange behavior is, if we observe the related keys num of a specific queue from living to deleted:

1. After the queue is created and being in a living state, the keys num is 6.
2. After the execution of obliterate(), the keys num is 0.
3. After 30 seconds of obliterate, there are two keys appear again, the keys num is 2. The 6 -> 0 shift is my expected behavior, but the 'two keys reappear' is unexpected. It can be reproduced from the code above.

In our RPC use case, there will be tons of reply queues being created and obliterated daily. If some of the keys reappear and keep exist (with no ttl), they will consume lots of unnecessary memory in our instance.

BTW, I have tried manually delete the two keys after each time I called queue.obliterate(), but they even 'self-recovers' after some seconds. This has totally confused me.

[manast]

3. After 30 seconds of obliterate, there are two keys appear again, the keys num is 2. The 6 -> 0 shift is my expected behavior, but the 'two keys reappear' is unexpected. It can be reproduced from the code above.

You need to close the worker (and the queue, but in this case, the worker is the one that is still running and creating the keys): https://api.docs.bullmq.io/classes/v5.Worker.html#close

[shan-mx]

@manast It worked. Thanks!