search

nestauk / daps_utils

Tools for setting up and running pipelines in a Data Analytics and Production System (DAPS).
0 stars 1 forks source link

[33] added force rerun #37

Closed jaklinger closed 3 years ago

jaklinger commented 3 years ago

Closes #33

jaklinger commented 3 years ago

@Jack-Vines if you try checking out this branch, you should now be able to do:

luigi --module something SomeCurateTask --force

to forcibly rerun the task, and

luigi --module something SomeCurateTask --force_upstream

to also forcibly rerun all children.

It changes the base of MetaflowTask and CurateTask to daps_utils.ForceableTask to make them "forceable"

I've not finished writing tests and checking for edge cases, but hopefully does the job. If you could check if you have any issues with it, that would be great!