Refactored the scripts to improve parallelism

PHACDataHub / data-mesh-ref-impl

Data Mesh Reference Implementation with standalone example use cases

Apache License 2.0

6 stars 3 forks source link

Refactored the scripts to improve parallelism #56

Closed simardeep1792 closed 10 months ago

simardeep1792 commented 10 months ago

Added GNU parallel to optimize the processing of file uploads. It enables concurrent processing of multiple files, leveraging available CPU cores to speed up the upload tasks.

simardeep1792 commented 10 months ago

Record uploading speed observations

BEFORE

Uploaded 7844 hospital records.
Uploaded 7823 practitioner records.
Uploaded 48404730 EHR records. Total time taken: 6067 seconds.

AFTER:

Uploaded 7833 hospital records.
Uploaded 7790 practitioner records.
Uploaded 41970487 EHR records. Total time taken: 1190 seconds.