[Question] Getting Processing, Dead and Message queues

[elribonazo]

First of all I must say I'm super happy with the performance of this Plugin. I would like to make an open question regarding redis-smq.

In the admin dashboard I'm working on, I'd like to have the hability of displaying the amount of running tasks, failed queues, etc.

I tried with a simple redis client, and trying to search on the following keys:

const keyTypes = {
    KEY_TYPE_MESSAGE_QUEUE: '1.1',
    KEY_TYPE_PROCESSING_QUEUE: '1.2',
    KEY_TYPE_DEAD_LETTER_QUEUE: '1.3',
    KEY_TYPE_MESSAGE_QUEUE_DELAYED: '1.4',
    KEY_TYPE_HEARTBEAT: '2.1',
    KEY_TYPE_GC_LOCK: '3.1',
    KEY_TYPE_GC_LOCK_TMP: '3.2',
    KEY_TYPE_RATE: '4',
    KEY_TYPE_RATE_INPUT: '4.1',
    KEY_TYPE_RATE_PROCESSING: '4.2',
    KEY_TYPE_RATE_ACKNOWLEDGED: '4.3',
    KEY_TYPE_RATE_UNACKNOWLEDGED: '4.4',
    KEY_TYPE_STATS_FRONTEND_LOCK: '5.1',
    KEY_TYPE_MESSAGE_QUEUES_INDEX: '6.1',
    KEY_TYPE_PROCESSING_QUEUES_INDEX: '6.2',
    KEY_TYPE_DEAD_LETTER_QUEUES_INDEX: '6.3',
    KEY_TYPE_SCHEDULER_LOCK: '7.1',
    KEY_TYPE_SCHEDULER_LOCK_TMP: '7.2',
};

Ideally I would like to understand how to search in this tree of keys, and avoid using a redis client. I could implement that part perfectly and would be happy to help but i'm lost on how finding the info.

[elribonazo]

Let me describe the process a bit better.

1. Get Dead queues: First I do lrange command ${keyTypes.KEY_TYPE_DEAD_LETTER_QUEUE}|${REDIS_QUEUE_NAME};

This seems to be working in production (but still don't know why local is not displaying them).

2. Getting the processing nodes, or consumers. I basically run hgetall command on ${keyTypes.KEY_TYPE_PROCESSING_QUEUE}|${REDIS_QUEUE_NAME} And for every row found I call llen to get the amount of tasks that its running.

3. Get the pending queue, exactly as the dead queue obviously changing the keys.

[elribonazo]

@weyoss What do you think? Are we pointing in the good direction? Calling the right commands and keys, seems we arent'

[weyoss]

@elribonazo hey! Sorry for my late reply.

First of all, I highly recommend you to consider RedisSMQ Monitor. It seems you are trying to reinvent the wheel.

RedisSMQ Monitor has been rewritten based on React and D3 and is now maintained separately as a stand-alone project. Please take a look here https://www.npmjs.com/package/redis-smq-monitor.

If RedisSMQ Monitor does not fit your needs then you should go your own way and make use of the real-time statistics provided through WebSocket.

The stats include all the data you need about the message queue (queues, consumers, producers, messages, failed messages, rates, etc...).

I hope my answer was helpful to you.

[elribonazo]

Hi weyoss, No worries about the late reply.

I'm sorry we finally had to move to another framework, and it was not due to the UI thing but our tasks were running on multiple instances on some cases. Our Redis task may be different to most common ones. For example we set tasks as paused, and sometimes a task completes very very quick because its paused and then, something is wrong with the locks because tasks run more than once.

Anyway, our goal was having the hability to submit failed tasks again, better checking the running, pending and scheduled tasks. We've gained a lot more control, by having:

1. Hability to see the logs of any task or any task iteration.
2. Builtin functionality to pause task runners.
3. Easy API to integrate this service into our existing dashboard.

[elribonazo]

I'd be happy to contribute, but if u really think this reinvents what u did it does not have many sense.

thanks anyway