[stencil] [C4GT] Queues with bull

[techsavvyash]

Add Queues with Bull to Stencil NestJS Project

Description

The task involves integrating Bull, a JavaScript library for managing job queues in Node.js, into Stencil. This addition will enable the project to efficiently handle and manage asynchronous tasks using job queues.

Goals

1. Bull Integration: Successfully integrate and equip Stencil with Bull.

2. Job Queue Implementation: Implement job queues using Bull for at least two distinct asynchronous tasks within the Stencil.

3. Concurrency Control: Configure and optimize Bull job queues for efficient concurrency control, balancing workload and resource utilization.

Acceptance Criteria

1. Bull Setup: Install and configure the Bull library in Stencil.

2. Job Queue Implementation: Create and integrate at least three different job queues for distinct asynchronous tasks in the project.

3. Concurrency Control Testing: Perform testing to ensure that concurrency control is effectively managed, preventing issues related to job processing overload or resource exhaustion.

4. Documentation Update: Provide comprehensive documentation on your work. Include it in the form of a READMe.md in the specific module you create. It will be added to the official Stencil Docs.

Checklist (To be filled by the developer)

• [ ] Install Bull library in the Stencil NestJS project.
• [ ] Configure Bull to work seamlessly within the project.
• [ ] Implement at least three distinct job queues for different asynchronous tasks.
• [ ] Implement error handling mechanisms for graceful error management.
• [ ] Test concurrency control to ensure efficient job processing.
• [ ] Update documentation with comprehensive information on Bull integration and job queue implementation details.

Product Name

Stencil

Organisation Name

SamagraX

Tech Skills Needed

TypeScript, APIs, Queues, Bull

Mentors

@RyanWalker277

Complexity

Hard

Domain

Category

Backend

Sub Category

APIs, Queues, Bull

[amansgith]

Hii @techsavvyash @RyanWalker277 I would like to give an attempt to this issue under your mentorship!! I have a few things to ask:

1. can you give me more info about the the async tasks for which the job queue is needed to be implemented.
2. Also is redis already connected to the project? As i know redis is required by bull to implement the job queues
3. Is bull needed to be installed in stencil or in any of its specific packages folder?
4. Can you provide some resource to understand what stencil is actually working to achieve and its workflow and understanding the project in depth

Thanks

[techsavvyash]

Hi @amansgith, Thanks for the detailed query, i'll try to provide some clarity on your queries:

1. Bull is to be used to enable the capability of asyn events in stencil so the implementation needs to be generic enough to provide an interface to implement async processing fast.
2. Redis containers can be found connected to the project in the current state of stencil.
3. Bull needs to be installed in the specific component that you will create to expose the interface to setup the tooling around bull, these components might be a controller, service, interceptor, module etc. which we might find required to help bootstrap bull faster in our stencil project.
4. Stencil is an Opinionated backend framework with a primary objective of reducing the developer time in setting up boilerplagte tooling which remains same across projects. The workflow is simple as of now, we have a common folder inside the repository which hosts the pre-defined components that we publish as a part of the @samagra-x/stencil package to enable the current functionalities that stencil supports, other than that we have the examples folder which demonstrates via working examples how to setup a particular feature that stencil provides.

Hope this helps provide clarity :)

[amansgith]

Thanks @techsavvyash this clears a lot!! i will proceed with it and will get back if anythings bugs me..

[techsavvyash]

Hey @amansgith , How is it going?

[amansgith]

Hi @techsavvyash I am currently learning about How to use queues.. it's my first time picking up such an issue where the software itself works as a boilerplate for other applications So it's going a little hard for me.. I Request if anyone else wants go ahead with the issue.. they are welcome!! Sorry for no progress.

[techsavvyash]

Hey @amansgith, Thanks for taking interest, and we do realise it might be something new for you. Please let me know in case you need some support.

[amansgith]

Hey @amansgith, Thanks for taking interest, and we do realise it might be something new for you. Please let me know in case you need some support.

Sure thing!! Thank you so much

[Yash-Sajwan24]

@techsavvyash I understand the problem, and I believe I have the necessary skills. Could I have the opportunity to work on this?

[techsavvyash]

@Yash-Sajwan24 please go right ahead!

[saksham-tomer]

Hello ,@techsavvyash I am working on this issue and I have a few queries. 1- I am thinking about setting the BullMq service inside the common/servics and making it a global module so that we can inject it as a dependency whrever required. 2- Additionally I was thinking about implementing the job queues for the file upload service to manage file upload and fallback 3- Can you give me some information about the environment variables, like the redis connection url etc. Thank you.

[Savio629]

I'll also give it a try

[Savio629]

@techsavvyash I tried intergrating bull in 04-monitoring sample with the help of a youtube tutorial to understand it and it works fine with the sample. I have attached a implementation video below:

(The implementation just shows transcoding a message (or a audio file) with the help of queue)

https://github.com/SamagraX-Stencil/stencil/assets/91362589/62f8e08f-698e-4f25-9859-4dc3c32980ca

Bull needs to be installed in the specific component that you will create to expose the interface to setup the tooling around bull, these components might be a controller, service, interceptor, module etc. which we might find required to help bootstrap bull faster in our stencil project.

So what I understand is we need to implement bull for these components such as a controller, service, interceptor, module etc. Example: Let's say under controller we need to implement Bull for processing uploaded files asynchronously... https://github.com/SamagraX-Stencil/stencil/blob/main/packages/common/src/controllers/file-upload.controller.ts#L20

[saksham-tomer]

@techsavvyash I tried intergrating bull in 04-monitoring sample with the help of a youtube tutorial to understand it and it works fine with the sample. I have attached a implementation video below:

(The implementation just shows transcoding a message (or a audio file) with the help of queue)

bull-04monitoring.mp4

Bull needs to be installed in the specific component that you will create to expose the interface to setup the tooling around bull, these components might be a controller, service, interceptor, module etc. which we might find required to help bootstrap bull faster in our stencil project.

So what I understand is we need to implement bull for these components such as a controller, service, interceptor, module etc. Example: Let's say under controller we need to implement Bull for processing uploaded files asynchronously... https://github.com/SamagraX-Stencil/stencil/blob/main/packages/common/src/controllers/file-upload.controller.ts#L20

I am having issues running the instance locally --> Error: Cannot find module 'C:\Users\saksh\Videos\stencilNew\stencil\stencil\packages\common\dist\main' at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1149:15) at Function.Module._load (node:internal/modules/cjs/loader:990:27) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:142:12) at node:internal/main/run_main_module:28:49 from where did you spin up the server ?

[Savio629]

Trying running it on gitpod

[saksham-tomer]

Trying running it on gitpod 👍 Thanks

[techsavvyash]

@Savio629 the video looks good. You are on the right track, what we need is to add the components (modules, services, etc.) required to setup the bull in a Nest/Stencil app, as first class citizens in the stencil packages. Take for example how we have added the Monitoring module, Temporal module etc.

[Savio629]

So what I understood is that I need to add job queues using bull for handling async tasks over this modules (Monitoring, file uploads,etc ), right? Or create a separate module(is this what you meant by first class citizen?) for bull for handling job queues and then integrating them inside/over the specified module? @techsavvyash

[techsavvyash]

Or create a separate module(is this what you meant by first class citizen?) for bull for handling job queues and then integrating them inside/over the specified module?

this is the correct assumption @Savio629. We need to be able to setup bull in any stencil app and then bootstrap it using interceptors or whatever other component we require to any api endpoints the user might want to.

[Savio629]

@techsavvyash Right now I was trying it with file-upload by updating the packages of file-upload Ref: https://medium.com/@raviyasas/how-to-use-bull-with-nestjs-18697f123e3c Am I on right track?

[techsavvyash]

What exactly are you trying to do by manipulating the file-upload package?

[ishqDehlvi]

@techsavvyash Is this issue opened??

[techsavvyash]

Yes @ishqDehlvi

[Soham-27]

Hello @techsavvyash, Is this issue still open? If so, could you please provide more details about it so that I can start working on it?

[vishalmaurya850]

Please assign this issue to me under C4GT

[Savio629]

Assigning to @Soham-27 @vishalmaurya850 You can also start working collaboratively on the issue

[techsavvyash]

Hey @Soham-27 @vishalmaurya850 Thanks for showing interest in the issue, please connect with @Savio629 to get more context around the issue, but an assignment would only once you have raised a draft PR for the issue.

[MohitNSamagra]

Hey

[Soham-27]

Hello @MohitNSamagra,

Are you involved in the Stencil project?

[jaanbaaz]

Hi

[JiyaGupta-cs]

@techsavvyash Please assign me this issue