These process are responsible for crawling and scraping gov.uk and populating the S3 bucket used to serve the mirrors. The have been replaced by a K8s cronjob called govuk-mirror-sync in the the govuk-helm-charts repo. The repository that contains the new application that crawls and scrapes the site is found at govuk-mirror.
These process are responsible for crawling and scraping gov.uk and populating the S3 bucket used to serve the mirrors. The have been replaced by a K8s cronjob called govuk-mirror-sync in the the govuk-helm-charts repo. The repository that contains the new application that crawls and scrapes the site is found at govuk-mirror.