Implement Job Queue for Message Extraction and Guild Members Fetching

[Behzad-rabiei]

Description:

To efficiently extract messages from channels and fetch the members list of guilds in our Discord bot, we need to implement a job queue system. This will allow us to process these tasks asynchronously and handle them in a distributed and scalable manner. We are considering two options, Bull and RabbitMQ, as potential job queue solutions.

Tasks to Implement:

1. Research and document Bull job queue

• Description: Conduct research on Bull, a job and task queue library in Node.js, and document its features, capabilities, and integration with our Discord bot.
• Research areas to cover:
  • Overview of Bull and its main features.
  • Job and task queue concepts and how Bull implements them.
  • How to set up and configure Bull in our project.
  • Integration with Discord.js and how to use Bull for message extraction and guild members fetching.
  • Best practices and recommendations for using Bull effectively.

2. Research and document BullMQ job queue

• Description: Conduct research on BullMQ, an enhanced version of Bull, which offers advanced features and improved performance, and document its features, capabilities, and integration with our Discord bot.
• Research areas to cover:
  • Overview of RabbitMQ and its main features.
  • Job and task queue concepts and how RabbitMQ implements them.
  • How to set up and configure RabbitMQ in our project.
  • Integration with Discord.js and how to use RabbitMQ for message extraction and guild members fetching.
  • Best practices and recommendations for using RabbitMQ effectively.

3. Evaluate and choose the appropriate job queue solution

• Description: Evaluate the research findings for Bull, BullMQ and RabbitMQ and choose the most suitable job queue solution based on our project requirements, scalability, ease of integration, and performance considerations.
• Decision criteria to consider:
  • Features and capabilities of Bull, BullMQ and RabbitMQ relevant to our use case.
  • Integration complexity with Discord.js and our existing codebase.
  • Scalability and performance requirements of our bot.
  • Community support, documentation, and ease of maintenance.

Additional Notes:

• Document the integration process and code examples for the chosen job queue solution to facilitate future maintenance and understanding.
• Consider the operational aspects and requirements for maintaining the chosen job queue solution (e.g., deployment, monitoring, error handling, etc.).
• Thoroughly test the job queue implementation to ensure proper message extraction and guild members fetching.

References:

• Bull Documentation: https://docs.bullmq.io/
• BullMQ Documentation: https://docs.bullmq.io/
• RabbitMQ Documentation: [https://www.rabbitmq.com/documentation.html]
• (https://www.rabbitmq.com/documentation.html)

[cyri113]

Why can we not use RabbitMQ work queues?

https://www.rabbitmq.com/tutorials/tutorial-two-python.html

I guess there might be a persistance issue:

Note on message persistence: Marking messages as persistent doesn't fully guarantee that a message won't be lost. Although it tells RabbitMQ to save the message to disk, there is still a short time window when RabbitMQ has accepted a message and hasn't saved it yet. Also, RabbitMQ doesn't do fsync(2) for every message -- it may be just saved to cache and not really written to the disk. The persistence guarantees aren't strong, but it's more than enough for our simple task queue. If you need a stronger guarantee then you can use publisher confirms.

If we implement another service, I recommend BullMQ over Bull (legacy).

In this case, would we couple the Redis instance to this service or make it available for all services? @mehrdadmms

I would also suggest using parent/child jobs, breaking each API request into a separate job.

[cyri113]

Tagging @amindadgar as this could impact DAOlytics too.

[cyri113]

@Behzad-rabiei was this completed? or should we move it to another iteration?