Port targets and collections from UKWA-UI to this project

[anjackson]

The proposal is to port all the static content from the UKWA-UI project over to this one, and augment it so Targets can be browsed as well as Collections.

For the Site Generation task:

• [ ] Ensure Targets correctly inherit their Open Access status.
• [ ] Update sitegen script to drop in Welsh pages (but leave in English) UNLESS a Welsh page already exists.
• [ ] Make the 14-digit Wayback datestamp available in the metadata
• [ ] Use file path references not IDs so .GetPage will work and we can refer to Targets from Collections and vice versa.
• [ ] Create sub-collections as a set of hierarchical folders and add Parent references.
• [ ] Use no-parent as indicator of what to put under /collection/
• [ ] Ensure Targets and Collections appear under their original WCT-era IDs (this needs an automated test and some documentation on how its done)

For the static site generator itself:

• [ ] Ensure Targets have pages and they link to Subjects and to (published) Collections they appear in.
  • Here's an example of the old TEP page
• [ ] Ensure Collections have pages an link to (published) Targets they contain.
• [ ] Copy over translations and pages.
• [ ] Reproduce the breadcrumb trail when browsing collections.
• [ ] Add the 'Search Within This Collection` feature.
• [ ] Add stats (X web sites in this collection, archived date of Target, etc.)
• [ ] Embed thumbnails/etc. from the (prototype) screenshot creation & caching API.
• [ ] Ensure NGINX setup will prefix with /ukwa/ and will strip/collapse/ignore the source trailer on the /ukwa/TYPE/ID/source/SSS... URIs. e.g. http://www.webarchive.org.uk/ukwa/target/26640414/source/subject => http://www.webarchive.org.uk/ukwa/target/26640414/

Other ideas

• [ ] Consider a PWID page (with explicit URL/Time/Archive/Scope parameters) for citation/resolution/bookmarking.
• [ ] Also produce JSON versions at suitable URLs?
• [ ] Build reports on hidden/blocked/etc. Targets and Collections for review.
• [ ] Also port the Open Data site into this project? Or otherwise theme consistently?

[anjackson]

Having battled the original site, it's turned out to be easier to just start over, adding the minor changes we need to a vanilla hugo-bootstrap (v.4) theme. This keeps everything nice and separate and makes it clear exactly what modifications are needed.

[anjackson]

Okay, this is going to be done in stages, so I'm breaking this down into separate tickets.

[anjackson]

The majority off this ticket relates to full porting of Targets and Collections to static site generation. This had been investigated in some detail, but generally found that because there are so very many Targets, the static-site approach becomes rather difficult to work with.

We can revisit this later.

[anjackson]

Looking at this again, because Target pages would be very useful.

The main blocker appears to be dealing with the scale. There are c. 80,000 Targets, so these take a while to generate or to pass through Hugo, so having these in the Docker build breaks the Netlify build. We need Netlify for the CMS workflow, so we can't just do everything like that.

So, for deployment, can we use the Netlify workflow and GitHub to manage the CMS pages, but integrate the targets and collections from W3ACT in some kind of ongoing update workflow.

i.e. ukwa-services just uses the NGINX image pointing at a set of static files. These are generated from some combination of this Git repo/image and targets/collection files. Airflow task used to update the site source files and then initiate the Hugo build.