The Format Registry Index v2.0

[anjackson]

The Format Aggregator, but make it better.

Make it more useful and usable.

• Go through the implementation, check it's working right.
• Go over history, document nature of failures and gaps, and outline ways to avoid and alert.
• Improve description of sources, provenance, and linking back to registries, esp. WikiData.
• Work with Yale to include software associated with formats, linking through to EaaSI if possible.
• Can we make it better for search engines, so people googling for file extensions might find it?
• Consider adding a Datasette view.
• [x] Consider shifting analysis and registry comparison out of the site and into separate notebook or similar.
• With permission, publicise how to engage with each registry.
• Align/re-use modules from https://exponentialdecay.co.uk/blog/pronom-release-statistics/
• Make it easier to get from e.g. TRiD to PRONOM IDs via WikiData, say?
• Compare Collections By Extensions: https://github.com/digipres/registries-of-practice-project/issues/22

Split this into DigiPres Workbench and keep the Format Registry Index simple?

• [ ] Move comparison/analysis stuff out of the Format Index and into DigiPres Workbench. See #21
• [ ] Add a Siegfried.JS option?
• [ ] Add a basic Publications plot from the SQLite file.

Here are some additional links/nodes on tools: #16

[anjackson]

Starting this in a fresh branch of sentinel, and beginning by tightening up the model handling a bit. Currently a big 'aggregate' script does most things with the updated downloads, but it's not very nice. I'm revising that, with the intention of:

• Separating the parts out. Make data transformation (don't call it 'normalisation') a distinct step.
• Not using weird globals during data mapping! Build up any catalogues/indexes as a second pass.
• Using an explict data model via Pydantic, rather than nested dictionaries.
• Pull code together into a neater package, foreging (FOrmat REGistry INdexinG)