Generate data set readme and datapackage

[kmcelwee]

• [x] Add frictionless to requirements.txt
• [x] Create datapackage.json. For each appropriate field...
  • [x] Add constraints (required, unique, min/max, pattern, etc.)
  • [x] Add description
  • [x] Add foreign key
  • [x] (now that we've rebased) Add interventions fields to datapackage
• [x] Write up readme with info from datapackage

Data cleaning

• [x] 6d0df83e02cbe3604dc31bf63115db53e1349918 Remove ink field
• [x] e2ea277c1eeedc599c6a89e591effda15c7bda96 Rename to references.csv
• [x] 75c01915b2ce430e76ee9fc6315442d5758740f5 Rename instances.csv to library.csv
• [x] 08dd770d0800ebc450c534cc1e6cd3d5faceec58 Remove redundant work_uri field
• [x] 643a56f935b6b4e2ee6e3841fc1807f4f364958d Remove short_title field
• [x] 784c70bf49c3300e9343d11f2f0590678ad1c87c Reintroduce full version of annotation IIIF images (🎉)
• [x] 5ce4a4c372c08593e000f9db0f17b3b7e43a863b Ensure multiple reference IDs documented in README and datapackage description
• [x] 4133a194695469cc04120ef1734b70fc5bfad1db Expand library filter so that it includes objects 243, 247 and 177

Notes

• Useful scripts from S&Co: https://github.com/Princeton-CDH/mep-django/tree/main/data-export-meta
• Generate rough draft of the datapackage using the command cd data; frictionless describe *.csv > datapackage.yaml (frictionless data observes local paths when running validate

[kmcelwee]

Questions

• I've left question marks throughout the description fields that I was unsure of
• Brackets on pages? Should I explain that?
• references.csv or de-la-grammatologie_references.csv?
• Work URI field has one value https://findingaids.princeton.edu/collections/RBD1/c43 and it’s a finding aid. It contains the same value as that in the catalog_uri field. Drop this?
• Short titles aren't necessarily shorter (?!)

URI	Title	Short Title
https://derridas-margins.princeton.edu/titles/155/	Lettre à M. de Saint-Germain	Lettre à M. de Saint-Germain du 26 février 1770
https://derridas-margins.princeton.edu/titles/153/	Lettre au prince de Würtemberg	Lettre au prince de Würtemberg du 10 novembre 1663
https://derridas-margins.princeton.edu/titles/124/	Oeuvres complètes	Oeuvres complètes de Karl Abraham
https://derridas-margins.princeton.edu/titles/158/	Oeuvres Complètes, tome II	Oeuvres complètes de J.-J. Rousseau, vol. II
https://derridas-margins.princeton.edu/titles/177/	Œuvres complètes, tome I	Oeuvres complètes de J.-J. Rousseau, vol. I
https://derridas-margins.princeton.edu/titles/230/	Œuvres complètes, tome III	Oeuvres complètes de J.-J. Rousseau, vol. III
https://derridas-margins.princeton.edu/titles/199/	Œuvres complètes, vol. VI	Oeuvres complètes de Franz Kafka

• References IDs aren't unique. Should they be?
  • This also led to a 404 when clicking the title on this page: https://derridas-margins.princeton.edu/references/de-la-grammatologie/26a/. When clicking on the image, you get to a non 404, but there it has zero references https://derridas-margins.princeton.edu/library/plato-phaedrus-1964/references/

reference_id	value_count
https://derridas-margins.princeton.edu/references/de-la-grammatologie/26a/	3
https://derridas-margins.princeton.edu/references/de-la-grammatologie/57a/	3
https://derridas-margins.princeton.edu/references/de-la-grammatologie/70b/	3
https://derridas-margins.princeton.edu/references/de-la-grammatologie/395b/	3
https://derridas-margins.princeton.edu/references/de-la-grammatologie/50c/	3
https://derridas-margins.princeton.edu/references/de-la-grammatologie/29b/	3
https://derridas-margins.princeton.edu/references/de-la-grammatologie/27c/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/39d/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/29d/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/29c/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/129b/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/57b/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/197a/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/85b/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/395a/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/132a/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/404c/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/52e/	2
https://derridas-margins.princeton.edu/references/de-la-grammatologie/26b/	2

• Two books are in annotations but not in instances: https://derridas-margins.princeton.edu/titles/243/, https://derridas-margins.princeton.edu/titles/247/

[rlskoeser]

@kmcelwee so glad you are flagging all of these!

• Brackets on pages? Should I explain that?

Please remind me what this is! 😆

KM: Here's some examples of values in our "pages" column for annotations.csv 🥴 p. 45 [572] p.257 Back flyleaf 1 verso.

---

• references.csv or de-la-grammatologie_references.csv?

Let's simplify and just call it references.

KM: ✅ added to todo list

---

Can we come up with a more meaningful name for the books? I think instances won't make sense outside our system.

KM: library.csv?

---

• Work URI field has one value https://findingaids.princeton.edu/collections/RBD1/c43 and it’s a finding aid. It contains the same value as that in the catalog_uri field. Drop this?

Yeah, drop it if it's redundant.

KM: ✅ added to todo list

---

• Short titles aren't necessarily shorter (?!)

Weird! I wonder if we're generating them wrong. From the examples you gave, I think we either drop the field or relabel it so the field name is more accurate. But if all that information is included in other fields, let's drop it.

KM: I don't think the short_title field adds information. I'm going to add that to our todo

---

References IDs aren't unique. Should they be?

Wow! This is interesting, and a good thing to catch. I think the team must have entered multiple copies of the same reference when they weren't sure which copy of a book to link it to. I don't think we should try to change that now, but just reflect the research that was done. It seems like the simplest thing would be to just allow it not to be unique and make sure we note this in the documentation. (Combination of reference id & title id should be unique, though, if it matters).

I see that the search results for these links returns only a single reference — that probably means the indexing wasn't configured properly to handle multiple versions of the same reference, which is probably one of the reasons this never got caught. I think it's ok to live with that; let's just try to document clearly & briefly what the situation is.

KM: Added documentation to-do above ✅

---

Two books are in annotations but not in instances

This is surprising to me, because team members were only supposed to document annotations that related to references in de la Grammatologie. I guess we should revise the queryset filter on the instance export to include them — seems like the simplest solution to me, but I'm open to suggestions.

KM: Added to todo list ✅

[kmcelwee]

@rlskoeser I edited my own responses in your comment, because I thought it would be more clear. Let me know if I should never do this again haha 😄

[rlskoeser]

Wow, that is a pretty non-obvious way to reply! 😆 🤯 (I know threaded replies can get a bit much, but yeah, please don't do that in future 🙂 )

I like library.csv !

I'll have to look into the page numbers to be sure (and if it's not obvious and there isn't any project documentation about it, then maybe we won't be sure!). My guess is that brackets are to indicate the page number is not actually printed. Maybe make me an asana task?

[kmcelwee]

@rlskoeser I think I've taken care of almost everything. I have some questions inline, and I'm sure you'll have comments. Here are my questions / takeaways from this PR. Looking forward to your feedback when I return!

• frictionlessData doesn’t seem to be able to check self-referencing foreign keys :/ Unfortunately, any failures are raised silently when the YAML is formatted incorrectly, so I don’t know if it’s me or them.
• I included readme_info.py, which I copy and pasted. It was very useful, thank you! Happy to remove it though if it’s unnecessary

[kmcelwee]

Further back-and-forth edits will take place with PRDS, but w/r/t development, this is done