search

OptimalBits / bull

Premium Queue package for handling distributed jobs and messages in NodeJS.
Other
15.55k stars 1.43k forks source link

Communication between main process and sandboxed processors #1067

Open alolis opened 6 years ago

alolis commented 6 years ago

Hello,

I am trying to figure out if there is a way to communicate from the processors with the main process (and vice versa) and send custom events. The reason is that I would like a processor to be able to emit events that the main process will be able to catch and do some side effects will the processor is still running. If everything was under the same process a custom EventEmitter would have been sufficient but unfortunately, this will not work with a sandboxed environment.

  1. From what I understand bull uses process.send and process.on in order to achieve communication between main process and the forked child processes. Is there a way to access the child.send method somehow from within the processor in order to send events to the parent while the main process waits with process.on?

  2. Would it be easier to refactor the job that is passed to the processor to be able to emit events via the underlying queue (which already inherits from EventEmitter) and to be able to do job.emit('custom-event', data) from within theprocessor and then on the main process queue.on('custom-event, handler)?

  3. Any other alternatives? I was thinking Redis pub/sub system since it's already there and I can use that for now if no other way via bull is possible at the moment.

Thank you for your time.

gcox commented 5 years ago

@alolis Could you accomplish what you're trying to do by creating another queue, eventQueue, that is consumed by your main process and which your processors publish events-as-jobs to?

...and do some side effects will the processor is still running.

Not sure if that means the processor needs to use the result of the side effects or if it just needs to trigger something that leads to those side effects. If the former, you should be able to listen to the eventQueue's events in your processor to determine when the event-job has been processed.

alolis commented 5 years ago

@gcox it's been a while since the question but I did go with (3) as I also mention here.

In my case the processor did not need the results of the side effects but the communication had to go two ways.

pwob commented 5 years ago

I kinda did a hack on this one and pass the custom event to job.progress method.