GREI 3: HDV Task - Improve OAI-PMH Harvesting

cmbz commented 7 months ago

Overview

"Our proposed project will significantly improve the widely-used Harvard Dataverse repository to better support NIH-funded research. A critical measure of the GREI program’s success is to standardize the discoverability across generalist repositories.

To help with this, we propose to improve the existing harvesting functionality in the Dataverse software based on the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) standard, and coordinate with other repository packaging standards to share or move metadata and data. Dataverse already supports the Bags as defined by the Research Data Alliance (RDA) Research Data Repository Interoperability Working Group.

Here we proposed to improve the support for Bags, test it for NIH-funded datasets, and explore and define the appropriate standard to use to move the metadata and data across generalist repositories This will help with a sustainable and succession plan - if one repository cannot support anymore a specific dataset, it will allow to easily move the dataset to another repository without losing any information about the dataset." (Source)

Issues

Spikes

[x] https://github.com/IQSS/dataverse-pm/issues/25
[x] https://github.com/IQSS/dataverse-pm/issues/85
[x] https://github.com/IQSS/dataverse.harvard.edu/issues/92
- https://help.hmdc.harvard.edu/Ticket/Display.html?id=287243 (SRDA)
- https://help.hmdc.harvard.edu/Ticket/Display.html?id=295542 (CIFOR)
- https://help.hmdc.harvard.edu/Ticket/Display.html?id=322153 (Borealis)
- https://help.hmdc.harvard.edu/Ticket/Display.html?id=329544 (DataverseNO)
- https://help.hmdc.harvard.edu/Ticket/Display.html?id=330637 (Roper Center)
[ ] https://github.com/IQSS/dataverse-pm/issues/100

Harvesting Issues (Year 3)

In Progress

[ ] https://github.com/IQSS/dataverse/issues/8129
[ ] https://github.com/IQSS/dataverse.harvard.edu/issues/243 - essentially done (we have harvested their metadata and their datasets are searchable by our users). we are keeping the issue open because we asked the devs at SRDA to improve something on their end, so this is for us to remember to check on the progress.
[x] https://github.com/IQSS/dataverse/issues/8243 - complete
[ ] https://github.com/IQSS/dataverse.harvard.edu/issues/204 - this is pending on Roper devs. fixing something on their end. We have reestablished contact and communicated what the issues are.
- https://help.hmdc.harvard.edu/Ticket/Display.html?id=330637
[ ] https://github.com/IQSS/dataverse.harvard.edu/issues/172 - ongoing effort
[ ] https://help.hmdc.harvard.edu/Ticket/Display.html?id=322153
[x] https://github.com/IQSS/dataverse/issues/10611
[ ] https://github.com/IQSS/dataverse/issues/10734

Complete

[x] https://github.com/IQSS/dataverse/issues/8629
[x] https://github.com/IQSS/dataverse/issues/3322
[x] https://github.com/IQSS/dataverse/issues/8005
[x] https://github.com/IQSS/dataverse.harvard.edu/issues/153
[x] https://github.com/IQSS/dataverse/issues/7624
[x] https://github.com/IQSS/dataverse.harvard.edu/issues/142
[x] https://github.com/IQSS/dataverse/issues/3437
[x] https://github.com/IQSS/dataverse/pull/10222
[x] https://github.com/IQSS/dataverse/issues/9992
[x] https://github.com/IQSS/dataverse/issues/10329
[x] https://github.com/IQSS/dataverse.harvard.edu/issues/251
[x] https://github.com/IQSS/dataverse/pull/10323 - almost done (currently in QA)
[x] https://github.com/IQSS/dataverse/issues/10254 - for the most part, the fix already exists (added to the prod. 6.1 patch - that's what made it possible to harvest from SRDA, next item on the list)

Pending

These issues will be sized and prioritized once the In Progress issues are closed.

Additional Harvesting Issues (Year 4)

These issues will be prioritized during GREI Year 4 Planning.

Resources

cmbz commented 7 months ago

Status: January 2024

Several issues relating to harvesting other repositories' metadata were resolved for Harvard Dataverse. A number of Dataverse harvesting-related issues and bugs were closed, see list:

Completed

In progress

cmbz commented 6 months ago

Status: February 2024

The following OAI-PMH harvesting improvements shown below were made this month.

Completed

cmbz commented 5 months ago

Status: March 2024

Meetings

A meeting was held on 2024/03/13 to:

Discuss current GREI harvesting issues defined here: https://github.com/IQSS/dataverse-pm/issues/171.
Discuss larger issues and spikes requiring substantive design and refactoring work and coordination
Define a plan of action for GREI Year 3 (and ideally, Year 4) to complete the known remaining harvesting issues and any new, in-scope issues that may arise.
Action Items

The following action items were identified:
[x] Create an ordered list of non-Dataverse repositories that need harvesting @jggautier
[x] Add additional issue/RT Tickets related to harvesting to the tracking issue @jggautier
[x] Create list of all Dataverse installations we should be harvesting from and identify best metadata format for each @jggautier
[ ] Re-create harvesting clients using most appropriate metadata format in order to harvest as much metadata as possible from each client, following draft policy @jggautier
[ ] Modify policy issue to reflect discussion: https://github.com/IQSS/dataverse-pm/issues/100 @scolapasta
[x] Create "repository harvest profile" for each non-Dataverse repository that indicates the type of harvesting-appropriate metadata exports it supports. Use these profiles to inform the harvesting process for each repository. @jggautier
[x] Order the existing Pending issues list in order of priority @landreev
[x] Identify the current, remaining harvesting issues in the dataverse repo and add them to the tracking issue in a new section @cmbz

Other Work Towards Improving Harvesting

@jggautier updated the GitHub issue at https://github.com/IQSS/dataverse/issues/5889 with a proposal for sending more metadata to DataCite

Completed

cmbz commented 5 months ago

Status: April 2024

@jggautier continued the March 2024 action item of re-creating harvesting clients using the most appropriate metadata format in order to harvest as much metadata as possible from each client. But he stopped due to technical issues being investigated, described at https://github.com/IQSS/dataverse/issues/10469, that are affecting harvesting.

cmbz commented 5 months ago

Status: June 2024

@jggautier emailed the contacts of all repositories we've been emailing about Harvard Dataverse harvesting from their repositories. In most cases he let them know that we continue to work on challenges that prevent Harvard Dataverse from harvesting from these repositories, and in a couple of cases, such as what's described in #243, he asked about the status of related work that they are doing so that Harvard Dataverse is able to harvest from them.

jggautier commented 2 months ago

@landreev, @cmbz and @scolapasta, today or sometime this week I'm thinking of following up with contacts of the repositories who've emailed our support email addresses about harvesting, to let them know that we're continuing to work on improvements related to how Dataverse indexes records, and some of those improvements may affect harvesting. Most of the emails sitting in my RT queue are harvesting related:

cmbz commented 2 months ago

@jggautier Great thanks! When you do, please update the June status comment, too https://github.com/IQSS/dataverse-pm/issues/171#issuecomment-2009648663

cmbz commented 1 month ago

Status: July 2024

Closed: https://github.com/IQSS/dataverse/issues/4859

cmbz commented 2 weeks ago

Status: August 2024

Updates

Figure out whether, or how to support the extended ISO 639-3 list of languages dataverse #8578
- We continued discussion and development work for extending the list of languages.
Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting dataverse #8129 and As a data repository, I need to harvest additional metadata in OAI_DC records dataverse #4176
- We continued discussion for improving the metadata included in Dublin Core metadata available when harvesting using OAI-PMH
Support for Importing and Exporting DCAT Metadata dataverse #1592
- We continued soliciting feedback from the Dataverse community about importing and exporting metadata using the DCAT metadata standard
Support MARC-XML for Export and Harvest dataverse #2721
- We continued soliciting feedback from the Dataverse community about importing and exporting metadata using the MARC-XML metadata standard

Harvesting Issues

Currently in QA: https://github.com/IQSS/dataverse/pull/10762
Currently in Review: https://github.com/IQSS/dataverse/pull/10737
Currently in This Sprint: https://github.com/IQSS/dataverse-pm/issues/318

IQSS / dataverse-pm