ukwa / ukwa-monitor

Dashboard and monitoring system for the UK Web Archive
0 stars 5 forks source link

Add a task to check that the W3ACT production database has been backed up #2

Closed anjackson closed 5 years ago

anjackson commented 7 years ago

We should have some checks that will throw out alerts if it looks like certain processes haven't run.

To start with, the W3ACT production database should be backed up onto HDFS once a day (by this task. A monitoring task could check if the output is present for yesterday, and raise an exception if not.

Currently, the backup path is:

/2_backups/crawler01/pulsefeprod_postgres_1/w3act.pgdump-20170714

and you should be able to check the task is complete using e.g.

yesterday = datetime.date.today() - datetime.timedelta(days=1)
target = BackupProductionW3ACTPostgres(date=yesterday)
if not target.complete():
    raise Exception("BLAH BLAH BLAH")

To do this, we need to add python-shepherd as a dependency for this project so it can inspect the tasks.

anjackson commented 5 years ago

Done via Prometheus.