search

covidgraph / motherlode

Pipeline for running all dataloader scripts for covidgraph in a controlled manner.
https://covidgraph.org
MIT License
3 stars 1 forks source link

parallelize datasource runs #10

Open motey opened 4 years ago

motey commented 4 years ago

Currently, all loaders run strictly sequential; loaders that don't depend on each other can be run in parallel

Therefore we maybe need to metric on how hardware hungry each dataloader is (Core count, Memory). As a start two new poperties in the registry will do it.