search

wmo-im / wis2-data-analysis

Apache License 2.0
0 stars 1 forks source link

start one process per batch #15

Open maaikelimper opened 12 months ago

maaikelimper commented 12 months ago

Chems mentioned that processing was slow. This PR introduces batch-processing on the queue, with each batch being processed in a new multiprocessing.Process.

(no bulk-insert in the DB introduced yet, as I suspect the download+processing is the main bottle neck atm)

celgarrai commented 12 months ago

Hey Maïke, I just integrated the proposed modifications. Now, the script runs faster. While dispatching to subprocesses, there was a blockage specifically at the database insertion. Indeed, there was a conflict between processes, particularly during the opening and closing of the connection (it's resolved now). I'm sharing with you the changes made on GitHub.

maaikelimper commented 12 months ago

Thanks, I could not test the changes I proposed, good to hear that you managed to resolve the conflict between the processes relating to opening and closing of database connections

celgarrai commented 12 months ago

thank you Maaïke