Add tasks to run under Airflow

[anjackson]

n.b. Some tasks need to be run on other services, but AirFlow can make the SSH connection dependencies nice and explicit (see example), and the remote task can still just run a Docker run command so we can manage the software distribution (note we might need a docker pull ukwa/ukwa-manage:latest as part of the remote script if were running latest).

Should also rationalise what's in:

• https://github.com/ukwa/ukwa-manage making sure to cover any fixes from the 2019-prod branch
• https://github.com/ukwa/ukwa-tasks
• https://github.com/ukwa/ukwa-hadoop-tasks

[x] Add https://github.com/epoch8/airflow-exporter so we can integrate with Prometheus.

Ingest

• [x] w3act_backup daily):
  • [x] Back-up W3ACT PostgreSQL database to HDFS as CSV and SQL.
  • [ ] Clean out older backups??? (optional)
• [ ] w3act_export (hourly):
  • [x] Update blocks/allows.txt/aclj and annotations.json.
  • [x] Update the Collection Solr.
  • [x] Update crawl feed/job specifications.
  • [x] python-w3act, w3act run_w3act-qa-check prod (currently, a weekly cron of 0 8 * * Mon ssh access@prod2 'sh /home/access/github/python-w3act/run_w3act-qa-check prod' >> /var/log/w3act-qa-check.log 2>&1)
  • [ ] GenerateW3ACTTitleExport (optional)
• [x] Analyse Crawl Logs/Run Document Harvester (hourly) see here
• [ ] Update Intranet Reports (GenerateHDFSReports, dead seeds, etc.) (optional) currently some in ukwa-manage here but also GenerateHDFSReports and ukwa-reports.

Crawl

• [x] crawl_launch (hourly):
  • [x] Launch crawls, based on crawl job specifications (hourly) (from here to crawlstreams).
• [x] crawl_warc_tidy (hourly): (to be part of ukwa-manage? OR h3cc?)
  • [ ] Close old open WARCs. (optional)
  • [x] Move WEBRENDER WARCs see here.
• [ ] Move WARCs and crawl logs to HDFS (hourly) ??? (optional) (new store command)
• [ ] Refillers, e.g. (optional)
  • [ ] post-process Kafka log or CDX and re-queue URLs that match certain criteria.
  • [ ] Run with CrawlCache and do screenshots/device emulation.

Management

• [x] Update TrackDB from HDFS:
  • [x] Update all HDFS daily, update WARCs locations hourly.
  • [ ] Metrics (generate metrics from TrackDB and push to the Prometheus push gateway) ??? OR stats_pusher.
• [ ] Back up TrackDB to HDFS. (optional)
• [ ] HDFS file hash job to TrackDB. (optional)
• [x] #102

Access

• [x] Update CDX Index with latest WARCs on HDFS, based on TrackDB (hourly) website/scripts/run-cdx-indexer.sh
• [ ] #63
• [ ] Back-up the Shine PostgreSQL database (daily)
• [x] Run the test suite (daily after the above updates?) and raise an alert if the website is misbehaving (optional)

[anjackson]

n.b. stats pusher

 docker run  -ti -e HTTPS_PROXY=$HTTPS_PROXY ukwa/monitor-stat-pusher:master python update_pushgateway_stats.py prod

[anjackson]

Going to close this old messy ticket and add some specific ones. See #103 and #104