Need documentation of OCFL implementation/subset

dmer commented 4 years ago

What is the current setup/configuration for OCFL?

What are the plans for OCFL?

DiegoPino commented 4 years ago

@dmer thanks. Guess who is getting this assigned? ME! Will work on this during the weekend 😄

DiegoPino commented 3 years ago

@dmer i transferred the issue into the https://github.com/esmero/strawberryfield because this is the implementer of such need.

A few points to discuss here (or more like questions)

1.- How much does your team do know/have evaluated/have been part of the specs of OCFL? 2.- How do you envision using this and when.

I spend some days and weeks really working around the original 0.1-0.4 specs initially when Archipelago started (as an idea, means even 6 months before the first code was pushed) and even thought of it as a valuable choice before going for S3. As you know Archipelago is Fedora 4,5 and 6 less and i have outspoken my roadmap concerns about the need of a Fedora in our environment many times (better said the no need for).

The specs, in their 1.0 Draft version clash a little bit with two important architectural pillars of Archipelago:

Optimal use of space (e.g is as highly redundant and Fedora3 was, permitting and enforcing duplication of binary media.
handles Assets (binaries) as part of the versioning.
Mimics a hierarchy which could not be true for the actual relationship between Objects
No support for multilingualism! (yes.. translationable assets are not taken in account as representations of the same thing)
Does not play well with other systems that are already in the pipeline of our preservation effors nor our access performance efforts. S3 is one of those because a lot of what S3 offers is a duplicated effort on OCFL

@giancarlobi @alliomeria in case you wonder what this is. Please give the specs a quick look. If in rush, this is how the V1 of a given Object would look like.. Pretty similar to our attempts on https://github.com/frictionlessdata/datapackage-php so that could be a good way of reusing this work too.

[object root]
    ├── 0=ocfl_object_1.0
    ├── inventory.json
    ├── inventory.json.sha512
    └── v1
        ├── inventory.json
        ├── inventory.json.sha512
        └── content
            └── file.txt

But for what is worth, the implementation is simple!

We already have an EventSubscriber that does the hashing, characterization and storing of Digital Objects in S3 (metadata) and binaries and a class that can generate the root/child structure that OFCL needs. Our subset is for sure not what you guys are looking for so you/we will need to go full blown.

Solutions:

Code/solutions delivered by us (need to check how this fits our roadmap, most likely not but we can work around that)
- A new storage plugin abstracting our current file deposit approach allowing many/other deposit systems.
- A new module that uses the plugin and extends our file persister service to implement full OCFL (but only under one scenario, not using all the NxN options OFCL provides, reason we need to have a use case. I can build a skeleton if that is what your Developers/customers need.
from you
- A use case and measurable developer commitment to test what we produce.
- documentation written and contributed by you

There is a second option, which is just an external listener connected to our S3 (webhook or SNS) that takes every deposited thing on our own storage and puts it into a OCLF structure. But this is all offline and does not interfer with the normal, daily needs of other users. What i mean is the effort of duplicating storage and structure is removed from Drupal.

This are my initial thoughts but will explore the latest draft a bit more and come back with more concrete actions.

dmer commented 3 years ago

@DiegoPino Thanks for the detailed response! My opinions are based on a fairly naive understanding of the wants and needs of the digital preservation folks and a lot of anecdotal evidence from different repository owners.

Many repository owners have a need for some form of "preservation" repository as part of their digital repository. If Archipelago allowed the easy addition of an OCFL repository, that would definitely be a plus for these folks.

That being said, I wouldn't put this at the top of my priority list for the roadmap - the basic installation and operation functions seem more important right now.

I'm interested in the second option you describe - this was sort of what I was imagining - that one could "turn on" the OCFL repo and it would be created as a secondary/backup storage. This has the effect of providing a backup in a transferable and platform independent format. I can only imagine that this would also be less of an overhead/resource burden on the system.

Looking forward to discussing this further.

DiegoPino commented 3 years ago

@dmer thanks. I'm going back now into OCFL to see if any of the latest spec elements add complexity of the opposite, simpler approach to the initial explorations and coding we did when all started. Will keep you informed in this thread but i do not see this as a complex feature to implement, just one that needs right planning so it does not become a bottle neck. Thanks!

DiegoPino commented 4 weeks ago

Out of scope of our already working storage system. Also we have a Hook that allows custom implementations if needed. Closing as not part of our roadmap

esmero / strawberryfield

Need documentation of OCFL implementation/subset #104