search

bundestag / gesetze-tools

Scripts to maintain German law git repository
GNU Lesser General Public License v3.0
114 stars 21 forks source link

🚀 [Feature] Multi-threaded processing #45

Open darkdragon-001 opened 3 years ago

darkdragon-001 commented 3 years ago

Problem to address Modern systems provide multiple cores which are unused when the scripts run in a single thread.

Feature description Process tasks concurrently to speed up processing of many laws significantly.

Implementation details Use queue module.

It should be used optionally with a command-line flag.

mk-pmb commented 1 year ago

If we can split workload accross different machines (virtual machines with one or very few cores), that would be even better in some scenarios. Maybe we can define some notion of todo range and then have each machine work on its assigned range? If needed we can run a todo scanner first that then suggests ranges. A 3rd-party script specific to the cloud infrastructure can then use those suggestions to spin up the appropriate VMs.