Source data model

[BeritJanssen]

Since we'll have sources from various platforms, we need to make sure that the interface can handle them. Necessary steps:

• [x] Get consortium input about what should be in the model
• [x] in progress Consolidate consortium responses
• [x] in progress Find suitable vocabularies
• [x] Form plan, request feedback
• [ ] Migrate
• [x] Adjust upload form
• [ ] Add fields on Elasticsearch

[jgonggrijp]

Related: #390.

[jgonggrijp]

This is the input I have gather so far on Slack. I consider this stage complete.

Field	Status quo	Gustavo (social media)	Claire (bibliography)	Alessio (crowdsourcing ontology, subset of schema:MediaObject)	Guillaume (scraped reviews)
title	required	desirable		schema:title	page_title (optional, generally same as book_name)
author	required	optional (anonymization)			author
editor	optional	optional
fulltext	required	necessary			text
screenshot/rendering	absent	necessary (eventually)
language	required	necessary		schema:inLanguage	language
source type	required, book/article/review/social media post/web content/other	necessary
publication date	required	necessary (tweet date)	optional, free-form text, "creation date" more appropriate for archival resources		date_text (free-form)
publisher	optional	optional/irrelevant	optional
URL	optional	optional (anonymization)		schema:identifier	url
repository	absent		desirable, free-form (usually archive, location, collection, call, fasc, folio)
format	absent			schema:encodingFormat
thumbnail	absent			schema:thumbnailURL
book author	absent				book_author_name
book title	absent				book_name
date of retrieval	absent				date_of_scrapping

Desirable sources to import from

• IRISA scraping tool
• UK-RED
• crowd contribution platform
• chatbot
• http://kvk.bibliothek.kit.edu/?digitalOnly=0&embedFulltitle=0&newTab=0

Regarding consolidation, some preliminary ideas:

• We might need three date fields: date of creation, date of publication and date of retrieval. RDF supports dynamic typing, so we can use xsd:date when possible and xsd:string otherwise.
• Several bits of input hint at future support for other formats than plain text. I'd really like to generalize this so that follow-up projects can add support for additional formats without needing to change the source data model. Plausible formats to support in the future include bitmap, PDF, Twitter JSON (possibly up to whole threads) and HTML.
• Dublin Core and Schema.org seem to be the most direct candidates for a foundation (which is not too different from what we already have). DC fits very well with the archival perspective. The one "catch" with this is that we are currently using DC for ownership and permission administration; we should probably move the latter to new terms in our own vocab or staff namespaces.

[jgonggrijp]

@JeltevanBoheemen I think we can merge #86 into this by just adding a public yes/no field.

[JeltevanBoheemen]

Good idea, I'll add it to the datamodel.