search

ErickWendel / parallelizing-nodejs-ops

Example of how to Migrate 1M items from MongoDB to Postgres in just a few minutes using Node.js child process
https://youtu.be/EnK8-x8L9TY
152 stars 76 forks source link

Using stream #2

Open WillianAgostini opened 6 months ago

WillianAgostini commented 6 months ago

Hi,

I'm sending some significant enhancements to optimize the script's performance. Here are the main changes implemented:

Streaming Query: Instead of using the skip() function to query the database, we're now leveraging the power of streaming to retrieve data as it's being read. This eliminates query overhead and provides a continuous flow of data in a single query.

Batch Insertion: To improve data insertion performance, we're now inserting data in batches.

Furthermore, we continue to apply the concepts of child processes. However, this time, I've implemented a dedicated child process solely for data querying. Once a list of items is accumulated, it's sent to other processes for parallel insertion, maximizing the benefits of parallelization already incorporated previously.

ErickWendel commented 6 months ago

Unfortunately, this is part of the source code provided in the video, I can't change the code as people would use it in case of any problem. Thanks a lot for the contibution!!