search

covidgraph / motherlode

Pipeline for running all dataloader scripts for covidgraph in a controlled manner.
https://covidgraph.org
MIT License
3 stars 1 forks source link

Better handling of one dataloader failing #4

Closed motey closed 4 years ago

motey commented 4 years ago

Current Status

At the moment the default behavior of motherlode is to cancel the whole import process if one dataloader fails (return code != 0)

Desired Status

It would be better to only cancel dependending following dataloders( dependencies are defined in defined in https://github.com/covidgraph/motherlode/blob/master/motherlode/data_sources_registry.py)

All independent dataloader tracks should just go on.

Profit

Unattended failing import processes would advance in progress and the rerun after fixing the failed dataloader would take less time.

frankschmitt commented 4 years ago

I'll take a stab at this. Stay tuned :-)

frankschmitt commented 4 years ago

@motey I've created a PR for this. Would you like to review it, or should I merge it myself?

motey commented 4 years ago

Pretty cool :) i will give it a test later or tomorrow

motey commented 4 years ago

just tested and looks good to me :) Thanks Frank!