DigiPres Publications Index v1.0 - the iPres Index

[anjackson]

Broadly following the Format Aggregator pattern.... Gather metadata and text from iPres proceedings. Make it easier for Google to find. Make it easy to search across. Start to think about how to link formal identifyers into the system, so we can find e.g. papers about GIFs.

This is a first iteration to demonstrate the idea, focussed on iPres proceedings.

• [x] link identifiers back to source record (idea from Micky)
• [x] Add full-text URL to DB citation_pdf_url (idea from Ed)
• [x] Add keywords (and any other gaps) to the list of full-text indexed fields and make this clear in the intro docs.

• 4

• [x] Fix deploy-preview links so they go straight to the page that's changed?
• [x] Make logo optional.
• [x] Show logo on conference page if available,
• [x] Current version re-downloads additional data or iPRES 2022 every time the final merge is run. This is not very efficient. Can it be altered so there is a separate fetch stage for iPRES 2022, which can later be extended to integrate OSF?
• [x] Current version is blatting files with the same names, e.g. https://www.digipres.org/publications/ipres/ipres-2023/papers/monitoring-file-format-obsolescence-in-repositories-an-applied-m.html only the presentation is available. That needs to be fixed, even if not via a full fix of patching together the presentation and paper into one records. Maybe construct some look-up tables in the first pass, then a join-and-drop phase at the end?

Writing up covered by digipres/registries-of-practice-project#7

Further work covered by #5

[anjackson]

Updated to link records back, and pull out institutions at least up to a point. Works pretty well in the browser, see this Datasette Lite view

[anjackson]

Looking at the other years. IDEALS turned out to be easier, as I can at least grab chunks of metadata via OAI-PMH and only have to futz with the configuration of that in order to have something useful. To my surprise, OSF is proving more difficult to work with, with two different API versions that don't seem to line up well, and with it being necessary to grab a 'tree' of different files.

[anjackson]

Now updated with some (slightly sketchy) 2022 and 2023 data in place, here.

[anjackson]

Okay, refactored a bit and used Ed's suggested citation_pdf_url trick to pull in the document URLs for 2023. Latest version now has separate landing page and direct document URLs. See here.

2022 data still a bit lacking, as OSF integration needs work.

[anjackson]

Probably need to spend a little time thinking about the tables/structures. e.g.

• Formats
• Publications
  • iPRES
• Policies

[anjackson]

Pretty clear, I think, that for now I'm going to have to patch up some of the metadata by hand, but I can at least get the basics in place from the repositories.

[anjackson]

Ah, if you add a _searchmode=raw you can do proper searches, like "this" OR "that". See here

[anjackson]

Okay, so now https://www.digipres.org/publications/ hosts a web-based version and points to the DB/Datasette version.

[anjackson]

Good enough for v1