Support running across two Hadoop clusters

ukwa / ukwa-services

Deployment configuration for all UKWA services stacks.

Apache License 2.0

5 stars 5 forks source link

Support running across two Hadoop clusters #49

Closed anjackson closed 2 years ago

anjackson commented 2 years ago

So we can upgrade/migrate as needed, we need to be able to run across two Hadoop clusters.

[x] Airflow #50
[x] https://github.com/ukwa/ukwa-warc-server/issues/8
[x] https://github.com/ukwa/ukwa-manage/issues/80
[x] https://github.com/ukwa/ukwa-manage/issues/82
[x] Add Airflow DAGs to run the above tools on both Hadoop services

anjackson commented 2 years ago

A DAG for CDX indexing has been created. Tested and running on h020, expected to work on h3 but awaiting tests.

anjackson commented 2 years ago

Pleasingly, the same indexer code now runs smoothly over h020 and h3 with little fuss.

anjackson commented 2 years ago

Okay, so we are now running over two clusters so I'm going to call this done. There are some minor issues outstanding but they'll be dealt with in other tickets.