Dataverse to HydraNorth

[pbinkley]

As a data librarian, I want to be able to ingest item-level metadata from Dataverse into the DAMS, so that Dataverse items are discoverable via the HydraNorth discovery layer while they are still accessed in Dataverse.

Part of epic #69

[pbinkley]

This may turn out to be too difficult to achieve without adding new models, but let's have a look and see if it can be done in this sprint. Out of scope: direct deposit of items without datastreams In scope:

• batch ingest of Dataverse items
• provide link back to Dataverse
• adjust display page for missing/new elements John's work on DDI extract is moving ahead, tracking community work - but still needs work (stylesheet is out of date re spec) We need to nail down our technical goals
• add new vocabularies
• fix stylesheet
• won't be in FOXML
• appropriate RDF mapping - this will determine whether we can use the current Sufia data model, or whether we need to create a new data model for the metadata - would do this after PCDM
• note that we can also just export DC from Dataverse - so could fall back on this
• need to determine the discovery needs for this content - get user stories from Chuck etc., researchers (part of summer usability project)
• note that the DDI model allows encoding of study variables in the metadata, which are useful for researchers - need to determine whether this level of discovery is needed in Hydra, or is Dataverse (new version) is where this happens - this is to be determined with stakeholders
• workflow: we won't allow deposit of datasets via Hydra - will need to enforce this non-Dataverse content (e.g. cow data - too big) - needs to be dealt with

Next steps:

• determine scope for Sept. - whether to implement DDI in RDF, or import DC - John will lead this, in consultation with Chuck and Larry
• trial migrations of DC - Weiwei

[pbinkley]

[Sharon:] Earlier discussion had concluded that we would allow the deposit of datasets into Hydra (ERA). If this is to be changed then the form will need to be adjusted accordingly and any datasets currently in ERA assessed for handling.

[johnhuck]

The product owner has determined that, for purposes of the September launch, the scope for this story pertains only to basic data discoverability, such as can be achieved by importing OAI_DC into HydraNorth from Dataverse.

Chuck and Larry will develop more detailed user stories around data search and discovery in Hydra to inform work on phase 2 (post-September) of this user story, which will revisit the question of whether or not to import DDI-RDF. We will look for gaps between these more granular user stories and the functionality of the DC imported in phase 1.

[johnhuck]

Sent a first draft of transformed oai_dc metadata sample to Weiwei on June 19th. The transformed metadata: -updates the namespaces of the dc: elements to dcterms: -maps values in dc:date to dcterms:created -maps values in dc:coverage to either dcterms:spatial or dcterms:temporal -maps dc:type (which contains free text values) to dcterms:description -assigns <dcterms:type>Dataset<dcterms:type> to all datasets to conform to established type values in HydraNorth -excludes dc:description elements that contain a formatted citation

Questions that remain:

• [ ] Should the introductory phrases that prefaced (and identified) the content in dc:coverage elements be retained for clarity or dropped? The phrases are: -Time Period Covered -Date of Collection -Country/Nation -Geographic Coverage -Geographic Unit -Geographic Bounding [i.e., bounding box]
• [ ] Should dcterms:publisher be added to the Hydra Data Dictionary?
• [ ] What is the ideal root XML element to use with the Hydra import script: the original element from the source metadata (<oai_dc:dc>)? <metadata>? <record>?
• [ ] Should <dcterms:spatial>Geographic Unit: [...]</dcterms:spatial> be retained or dropped? Examples of values in this field are "Province" and "Census subdivision".
• [ ] Should any attempt be made to clean up the html markup in the description field, which appears to have been introduced by Dataverse users, acting in non-conformance with submission guidelines, seeking to create unordered lists of contents.

Other issues: At least one of the oai_dc records (10205) wouldn't resolve with its DOI. The metadata was very thin, which makes me wonder if it is an unreleased study or was withdrawn. When we start to work with a more current metadata extract, we should check to see if this issue resolves itself.

[weiweishi]

Migration script is in dataverse-migration branch for demoing, and reviewing the work. related questions/issues #452 #451 #450 #449 #448. objects migrated are in Plano.

Will leave it to @pbinkley and @johnhuck to decide if this ticket can be closed.

[pgwillia]

• [ ] html markup showing up in description
• [ ] review rdf in Fedora

[pbinkley]

Sonya and John will chase down answers to outstanding questions and we'll come up with stories to reflect the remaining work. Tricia will do a PR of the current state of this story, and we'll close it.