ukwa / ukwa-services

Deployment configuration for all UKWA services stacks.
Apache License 2.0
4 stars 5 forks source link

Crawl maintenance tools for re-crawls #116

Open anjackson opened 1 year ago

anjackson commented 1 year ago

Sometimes crawls fail for transient reasons, showing as e.g. -61 status code (cloud hosting service interruptions), and possibly -2 and -3 connection failures. It's be good to have a way to scan Kafka logs or other sources for this and enqueue a re-crawl.