search

wilk / microjob

A tiny wrapper for turning Node.js worker threads into easy-to-use routines for heavy CPU loads.
https://wilk.github.io/microjob/
MIT License
2.02k stars 47 forks source link

worker pool with clustering #10

Closed syberkitten closed 5 years ago

syberkitten commented 6 years ago

When the thread pool feature will be added, a question that comes to mind, can the thread pool be shared between multiple forked workers working in a cluster.

for example having 4 cpus, each running a forked node process (using native nodejs clustering) the worker pool should be for all 4 cpus and not per cpu, otherwise it will consume too much memory and resources.

best would be to configure a pool either local per cpu, or global, per machine (one pool for all cpus)

What's your take on this?

wilk commented 6 years ago

@syberkitten thread pool is already implemented. This is an interesting topic and I think we should investigate more.

It could be a cool feature for future versions 💪

wilk commented 5 years ago

I thought about this feature and I don't think it's feasible. The problem is that the worker thread spawns threads from a single process and you cannot have threads shared between different processes. Basically, a process cannot ask to another process to provide one of its thread because threads share the memory with the parent process and the operating system is very strictly on this: every process has its own private memory address allocation.

I'm closing it but feel free to re-open if you have evidence I'm wrong. I'll be happy to implement it, if possible ;)