I am developing a Node.js application that involves heavy data processing with three distinct services, each managed by a separate worker pool. Here's the breakdown of the setup:
ServiceX and ServiceY: Each runs with a single worker (serviceX.js and serviceY.js).
ServiceZ: This is the core of my application, handling extensive data structures (like an array of 10k objects) and data processing, running with N workers (serviceZ.js).
Current Architecture
In ServiceZ, a task involves checking for duplicates in a database where a specific column is not null. Additionally, I need to handle duplicates where the column is null in returned data to optimize processing and database updates. Given the intensity of these operations, I am considering using a nested worker pool within ServiceZ.
Question
Is it practical to initiate a new worker pool within an existing worker's task in ServiceZ? This new pool would potentially launch single worker instances dynamically to handle specific heavy operations like duplicate checking.
Concerns
Performance: Could nesting worker pools lead to significant performance hits due to overhead or complex resource management?
Best Practices: Is this approach advisable in terms of architectural best practices, or are there more efficient methods to manage such heavy operations without risking call stack blocks?
Description
I am developing a Node.js application that involves heavy data processing with three distinct services, each managed by a separate worker pool. Here's the breakdown of the setup:
serviceX.jsandserviceY.js).serviceZ.js).Current Architecture
In
ServiceZ, a task involves checking for duplicates in a database where a specific column is not null. Additionally, I need to handle duplicates where the column is null in returned data to optimize processing and database updates. Given the intensity of these operations, I am considering using a nested worker pool withinServiceZ.Question
Is it practical to initiate a new worker pool within an existing worker's task in
ServiceZ? This new pool would potentially launch single worker instances dynamically to handle specific heavy operations like duplicate checking.Concerns