CLAW Harvest Use Cases, Features, etc.

[uconnjeustis]

It would be great to have the following features with harvests going forward:

• Deletion support.
• Ability to have oai sets based on namespaces. This would greatly facilitate sharing data.

[alehman-loc]

@uconnjeustis Are you talking about the Islandora OAI Module? Or do you have another way that this works?

[uconnjeustis]

@amandarl Yea... sorry. This is in reference to the Islandora OAI Module. I thought it would be great to add this as it is related to metadata. I know that CLAW has included deletion support already!

[ruebot]

I know that CLAW has included deletion support already!

Where is that mentioned?

[ruebot]

Ability to have oai sets based on namespaces. This would greatly facilitate sharing data.

Namespaces are not a thing in Fedora 4. I'd definitely check out this use case: https://github.com/Islandora-CLAW/CLAW/issues/396

[uconnjeustis]

@ruebot I thought I remember hearing something about deletion support at the Fedora 4 camp in NYC. It was in the discussion about the tombstones. I don't know if there's any documentation or formal mention of it.

Thanks for bringing that up @ruebot. The use case that I'm thinking of is slightly different in that we work with different institutions where we need to distinguish their content in some way. We use namespaces that are assigned to each institution. Some namespaces designate content that shouldn't be harvested as well. In future, we would like to be able to provide these institutions with the ability to harvest their content (oai set). In the fedora3 view this would be a namespace set.

@ruebot Would it make sense to develop this a little using the use case template and add it to Issue 396?

[ruebot]

Fedora 4 does tombstoning. This would allow for a future, as yet designed, Islandora CLAW OAI-PMH module allow for OAI-PMH deletion support. Deletion support in CLAW is outlined in the MVP -- see: Delete a Resource -- and will use the default Fedora 4 behaviour of tombstoning.

I think the best course of action would be to arrange some community conversations about what the community would like to see w/r/t to OAI-PMH support in a Islandora. Then that would allow us to pull together fuller use cases, identify stakeholders, and resources to implement this.

Namespaces are a separate, but I can see how it is related, issue. If you have thoughts or opinions there, I would definitely weigh in on https://github.com/Islandora-CLAW/CLAW/issues/396, and would also suggest the same thing as about with a community led discussion, and identify stakeholders and resources to make it happen.

[mrmiguez]

Something else that would be useful:

Serializing JSON-LD to harvesters as well as XML.

[uconnjeustis]

@mrmiguez Are you thinking of aggregators like DPLA?

[mrmiguez]

Yes Jennifer. I've been working on serializing our MODS to JSON-LD for DPLA, and I just heard from Bryan that JSON-LD will be used by CLAW for internal data storage. It'd be nice to have the option to just dump that out, as well as building our friendly XML standards for serialization.

On Fri, Jan 6, 2017 at 2:40 PM, Jennifer notifications@github.com wrote:

@mrmiguez https://github.com/mrmiguez Are you thinking of aggregators like DPLA?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/islandora-interest-groups/Islandora-Metadata-Interest-Group/issues/38#issuecomment-270987238, or mute the thread https://github.com/notifications/unsubscribe-auth/AJlh5J53xpjjkKdfjxVEnrFoNnkK6ns7ks5rPpjDgaJpZM4LQ4fo .

[uconnjeustis]

This might be dumb question. But can DPLA take json-LD?

Continuing with OAI harvests, I would also like the ability to select what constitutes a set. I could see some uses to be able to select only top level collections or all collections.

[mrmiguez]

They can! Though I don't think anyone is delivering it yet. During the service hub application process, we were encouraged to try giving them JSON-LD. One of the (dis)advantages of being late the the service hub game?

[uconnjeustis]

This is brand new and definitely something to investigate. I'm thinking of gathering some of these ideas in a series of use cases. We have the MODS to RDF discussion today (1/9/2017) at the MIG. This could be a great discussion for February.

[uconnjeustis]

I wanted to sum up some of the features for harvesting in this user story. I'd like to bring this story to the attention of the claw group. If you'd like more features, add to the remarks or enhance the story.

Title (Goal)	OAI-PMH Harvest
Primary Actor	Repository Administrator
Scope	access, configuration, configuration
Level	TBD
Story	As a repository administrator, I would like to have a configurable OAI-PHM harvest that would expose metadata in multiple formats to either users or data harvesters.

Remarks on configuration

• Name of repository
• Path to the repository
• Repository unique identifier
• Administrator email
• Maximum response size
• Expiration time
• Exclude types of content
• Ability to have no OAI sets or have sets by types of content and/or collections
• Ability to append thumnail identifier to any request (whether DC, MODS, etc.)
• Support deletion
• Ability to configure harvest for Dublin Core, MODS, JSON-LD