Digitization Folk Fest Ingest

[pgwillia]

Is your feature request related to a problem? Please describe. The first set of digitization content will be the Folk Fest Programs. This set was chosen because it is small (there are 36) and well described.

• [x] ~Add has_one_attached :historical_archive to Book Model which takes the full archive for all directories relating to that object and archive it in case it's needed. Will not appear in the UI~ Remove has_one_attached :historical_archive from Book Model and replace with metadata attribute relators:rps (http://id.loc.gov/vocabulary/relators/rps) to link to the complete AIP stored elsewhere.
• [x] fulltext will be extracted from the ALTO xml. Should be easy to access in a join table but not the main object table so that results stay efficient.
• [x] Attach one high quality PDF

Initial ingest assumes that we'll push the content (triples csv, pdf, alto) to the server for ingest from a rake script.

Follows #2010

[mbarnett]

– There is a spreadsheet somewhere that maps the folkfest stuff to Swift IDs. I'll try to chase that down, or Sarah knows the details.

– We talked a bit about how we'll want to deal with Preservation/AIPs, and previous archival efforts, and the cheap interim solution I landed on was that we'll just take the tarred (or zipped or whatever the package is) copy of whatever we pull out of Swift, extend the book (and future) models w/ has_one :historical_archive, and attached the full thing to the model. The historical archive will never be loaded or made available for UI purposes and we'll have no reason to ever want to load it, so the attachment's only purpose is, in the future, somebody can find the old copy of the data.

– We're just looking to attach one big high quality PDF of the work for things like this and newspapers, not per-page images, and I'm unsure if that's what we'll get out of Swift or if this will require some preprocessing.

– The fulltext is in a metadata file whose name I forget right now. We'll have to find a parser for it if one exists, and figure out how to store the data – I'm thinking as an attachment if the parsing isn't incredibly complex, or in a text field on a not-normally-loaded join table if we need to heavily pre-process the data upfront and keep the indexable copy somewhere we can get at it easily? The reason not to store the full text on the table itself is that that will make reading the row back from the DB for search results and general metadata viewing way to costly – we'll only ever need the fulltext data for (re-)indexing (I guess this may argue for the pre-process into join table approach bc we'll need to reindex that data on save? Thinking ahead, we don't want to have to pull the text file from the Cloud just to re-save a file, so likely we want that in the DB somewhere, in a cheap-to-reindex form)

[sarahseverson]

• Here is the spreadsheet with the Peel numbers and matching Swift NOIDS. I've asked Tianyu to grab the file packages so we can see what is inside when we meet next Thursday.

[mbarnett]

Thanks!

[sarahseverson]

FYI: Here is a Google Drive folder with all the Folk Fest items from SWIFT - it's big at 11 GB but we pulled everything so you could see what's there. Have fun with the files!

[pgwillia]

FYI: Here is a Google Drive folder with all the Folk Fest items from SWIFT - it's big at 11 GB but we pulled everything so you could see what's there. Have fun with the files!

@sarahseverson I made an assumption that this was exactly how the material was in Swift: In directories with the noid label. But there's some indication in Matt's note that the content might have come compressed

take the tarred (or zipped or whatever the package is) copy of whatever we pull out of Swift

What format was it in? I've assumed that we should store it as *.tar.gz, does that makes sense?

[pgwillia]

Peel preservation examples was prepared by @kgood to show some of the differences in structures in Swift.

[pgwillia]

@kgood @sarahseverson @henryzhang87 and I met today. We discussed some questions I had:

1. Will a spreadsheet like https://docs.google.com/spreadsheets/d/1U8GckSd-tTaGlBk-TllPl1cd80h8PSQqEdKXUuoCNlI/edit?usp=sharing be created for each batch mapping "Code" to noid?

Yes, though there might be some differences; newspapers might be the biggest exception.

2. Is it possible to access Swift from an application server where the ingest happens? How will we do authorization/authentication? Or who/when/how will we stage this material? For the FolkFest materials Sarah and Tianyu moved 11 GB to https://drive.google.com/drive/folders/17ckV-is6Mh_ZA_tPQIHjZPwayxz_T6BN

We came up with three ideas

1. Using geoffry as a staging environment
2. pmpy
3. Open Stack Swift read only access from the application server

ACTION: Henry will set Tricia up on geoffy with access to the OpenStack Swift API

My hope is that we will be able to do # 3 when this gets rolling

3. What format does a download from swift come in? I'm assuming that it's compressed?

Swift downloads directories without compression exactly as Sarah/Tianyu provided above i.e. swift download peel --prefix=<noid prefix>

noid/type [tiff/jpeg/etc]/1.tar

Some swift directories won't have pdfs: like the Images will be tiff or jpeg

4. What archival format is appropriate for our attached historical archive?

ACTION: Kenton will follow up with Tricia about this.

[sarahseverson]

re: 1 - here is an example of a spreadsheet for a Newspaper upload to IA where we have a NOID, code (three letters) and some individual item-level metadata (year, month, day, pages)

[pgwillia]

We had further discussion about the historical_archive/AIP this morning.

• we're worried about storage costs in the cloud
• don't know if we'll be able to get to the point where we will delete the duplicate packages
• looking at olark?
• will look at connecting/linking both instances with metadata so that it's possible to do updates/evergreening if the content moves

Next steps: @sfarnel will look at the appropriate metadata fields and update this issue @pgwillia will modify the model to replace the attached historical_archive with a metadata property

[sfarnel]

Based on discussion with @kgood @pgwillia @sarahseverson

Recording the metadata aspect of the discussion here so that it is captured and can then be incorporated into our AIP specifications for PMPY. The new AIP generated on ingest will include in the metadata a link to the original (i.e., legacy) AIP in the form of a PREMIS relationship type property. The metadata should be: relSubType:sup [PID of legacy AIP], where the PID is likely to be UUID (as this is how we are naming AIPs) and, for reference, the full URI for the property is https://id.loc.gov/vocabulary/preservation/relationshipSubType/sup (which indicates that the new AIP supersedes the original or legacy AIP).

We also discussed the desire for recording the container in which the original or legacy AIP is located, which can also be captured through a metadata property: relators:rps [name of storage container], where the name of the storage container could be something like OpenStack/Swift, and the full URI for the property is http://id.loc.gov/vocabulary/relators/rps (which indicates that it is a repository for the object)