We should have some checks that will throw out alerts if it looks like certain processes haven't run.
To start with, the W3ACT production database should be backed up onto HDFS once a day (by this task. A monitoring task could check if the output is present for yesterday, and raise an exception if not.
We should have some checks that will throw out alerts if it looks like certain processes haven't run.
To start with, the W3ACT production database should be backed up onto HDFS once a day (by this task. A monitoring task could check if the output is present for yesterday, and raise an exception if not.
Currently, the backup path is:
and you should be able to check the task is complete using e.g.
To do this, we need to add
python-shepherd
as a dependency for this project so it can inspect the tasks.