Preparing to release the website stack

[anjackson]

Specific questions to consider:

• [x] Does the README make sense to you? (n.b. added sections on pywb access lists and other tasks associated with the stack).
• [x] Does the access-list update script make sense to you?
• [x] How should we manage the blocks.aclj file and it's deployment?

To Do

• [x] ANJ to update the blocks file from GitLab if it's simple enough to do so.
• [x] ANJ resolve pull request to merge anjackson/ukwa-manage:trackdb-lib into ukwa/ukwa-manage and update these scripts accordingly.
• [x] ANJ Add scripts to migrate TrackDB data from one instance to another.

[anjackson]

Hi @GilHoggarth I've made you a reviewer of this pull request, and added specific questions above that I'd like you to consider.

[anjackson]

• On update scripts:
  • The idea is that the update script should rely on environment variables set per deployment context. This is not set up right yet. I'll try to do it how I think you do it.
  • The Collections Solr update process will be very similar to the access list update process, but it's not done yet.
  • The scripts should be configurable via environment variables -- I'll try to copy how you've done it.
  • Yes, the anjackson/ukwa-manage fork should be resolved before deploying to beta/prod.
• On blocks.aclj deployment:
  • Are you sure you want to manually deploy the blocks.aclj file (which we'll need to keep synchronised across DEV/BETA/PROD and RR/non-RR services)? I can pretty easily set up a task to pull it from GitLab, for example.

[GilHoggarth]

In scripts/collections-solr/ there's a typo ('reqeusts') in get-prod-collections.py. And this script now seems to do both steps, its and populate-docker-collections.py. Would be good to do either all in one or in two steps.

[GilHoggarth]

The shine database first step of 'docker service rm website_shine' ran without complaint (from docker I mean) but the shine database still existed immediately afterwards, as shown by 'source list-db.sh'. Consequently I couldn't run the following steps, create-db.sh and create-user.sh. After this, download-shine-db-dump.sh and restore-shine-db-from-dump.sh still worked, though I'm unaware of the shine db state given I must have restored onto an existing database.

[GilHoggarth]

Issues with pywb, logs "--- no python application found, check your startup logs for errors ---". Believed to be related to configuration settings for pywb, concerning pywb-ca.pem (which doesn't exist at /mnt/nfs/config/gitlab/ukwa-services-env/access/pywb/pywb-ca.pem as configured within ~/github/ukwa-services/access/website/config/pywb/config.yaml

[anjackson]

n.b. example of how to run the Solr indexer under Docker (not ready to use yet!)

https://github.com/anjackson/ukwa-services/blob/trackdb-indexer-workflow/tasks/run-solr-indexer.sh