esmero / format_strawberryfield

Set of Display formatters, extractors and utils to make Strawberry field data glow
GNU Lesser General Public License v3.0
6 stars 9 forks source link

OAI for Archipelago #106

Open alliomeria opened 3 years ago

alliomeria commented 3 years ago

What?

Check another box off the Roadmap!

DiegoPino commented 3 years ago

Nice. Ok, since this is archipelago, tiny little islands where we build stuff everyday using twigs and scrap metal (@alliomeria et Al) so we are going to do things differently that the contributed D8 OAI modules in the wild (here is my plan, lurkers take note!) Ingredientes:

You can configure Arguments that are allowed and their format (GET API Builder!):

  1. Arguments can affect 4 different things: A. Can pass (map) from the API endpoint URL via GET to the View internally (e.g when calling a for a. Set) B. Can pass(map) from the API endpoint URL via GET and modify which actual Metadata endpoint is called for each record (IIIF, MODS, etc). Also only ones matching the Primary response type of this API endpoint will be allowed to be set as options. (nice!) B. Can pass(map) from the API endpoint URL via GET and feed(injected into) the actual metadata endpoint Twig source/Metadata Display entity (as $context) for cool dynamic rendering and modification and stuff of metadata
  2. Allow us also to generate a Wrapper (Also Twig template) Response. All that goes around the actual data.
  3. Caching works great on these (because caching will not be applied to the response itself (each Node) but to the actual Node rendering. So add a node and you get immediate response.
  4. Permissions are of course enforced and we can dynamically also also roles/assignment as we do for Form Modes (plugin derivers, since this are config entities)

This also solves of course any issues with "Uh, Oh, I need ETDS, can someone code it?" Twig template + add to the config, map the value. Done.

@alliomeria @giancarlobi how do you feel about this?

And other uses (wait for it)

Final crazy idea (which we already discussed with Giancarlo at some moment in the past but I can not find which issue) https://apisyouwonthate.com/blog/openapi-v31-and-json-schema-2019-09

e.g of an API described via openapi

parameters:
  - in: query
    name: limit
    schema:
      type: integer
      maximum: 50
    examples:       # Multiple examples
      zero:         # Distinct name
        value: 0    # Example value
        summary: A sample limit value  # Optional description
      max: # Distinct name
        value: 50   # Example value
        summary: A sample limit value  # Optional description

My crazy idea is quite simple. The configuration entity will be setup in such a way that the output of one of these will have close shape to the OpenAPI specs to define the arguments (in:query) and what they do.

That way you can even invent your own API. And done.

Good night.

DiegoPino commented 3 years ago

Another note: This is the custom Views Display (for data) i created that can resolve Entities https://github.com/esmero/webform_strawberryfield/blob/master/src/Plugin/views/display/EntityReference.php

it is one option. Now that we are natively indexing parent/child relationships as EntityReferences in Solr, we may not even need this any more.

DiegoPino commented 3 years ago

@alliomeria @pcambra @giancarlobi ok, I'm on this now. Always the "frame" around the code is the most expensive part

A few notes for myself where I will start (repeating my issue body really)

  1. This implementation will use at least 1 twig template/Metadata displays but normally really 2 as config option. a. First template is the Protocol response wrapper. E.g in the case of a OAI will contain the global XML that surrounds individual records b. Each record may be rendered individually by a second template. This way we have separation of "expected" API response v/s individual records (a single NODE)

  2. The API will have a single (for NOW) view as a setting that will generate/used for generating a listing of resources based on arguments (exposed filters).

  3. Configurable Arguments of the API may either map to:

    • View Exposed Filter: Count of records, additional arguments we may allow the user to pass. Each argument will be Strictly typed and validated
    • A Metadata Display Selection (Mapping Form Element): Certain arguments may/no trigger a change on the Secondary (plural) twig templates, the ones per record.
    • Data passed directly to the master: Twig Template and also passed to each record. Also Strictly typed and validated

Not sure I need anything else?

DiegoPino commented 3 years ago

This issue here is growing. I just learned that OAI is a bad bad API. In case a given ID was not found instead of returning a 404 HTTP code it returns a 200! found with a body message. So not much of a real REST API at all.

E.g

dpino@Diegos-Mac-mini metro % curl -iv "http://export.arxiv.org/oai2?verb=GetRecord&identifier=oai:arXiv.org:quant-ph/02131001&metadataPrefix=oai_dc"
*   Trying 128.84.21.203...
* TCP_NODELAY set
* Connected to export.arxiv.org (128.84.21.203) port 80 (#0)
> GET /oai2?verb=GetRecord&identifier=oai:arXiv.org:quant-ph/02131001&metadataPrefix=oai_dc HTTP/1.1
> Host: export.arxiv.org
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 04 Feb 2021 19:26:48 GMT
Date: Thu, 04 Feb 2021 19:26:48 GMT
< Server: Apache
Server: Apache
< Vary: Accept-Encoding,User-Agent
Vary: Accept-Encoding,User-Agent
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Content-Type: text/xml
Content-Type: text/xml

< 
<?xml version="1.0" encoding="UTF-8"?>
<OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd">
<responseDate>2021-02-04T19:26:49Z</responseDate>
<request>http://export.arxiv.org/oai2</request>
<error code="badArgument">Found illegal duplicate of argument 'amp'.</error>
<error code="badArgument">Verb 'GetRecord', illegal argument 'amp'.</error>
</OAI-PMH>
* Connection #0 to host export.arxiv.org left intact
* Closing connection 0

The specs are terrible http://www.openarchives.org/OAI/openarchivesprotocol.html#ProtocolMessages Which means to have a flexible API builder in archipelago we may need to allow people to decide on how to respond to exceptions... I was hoping to have to avoid writing body for things that do not check.

dmer commented 3 years ago

friendly request for status update on this - our project has a need to provide an OAI feed to its main harvester/aggregator (Calisphere) by the time we go live (hopefully in September). We are available and eager to do extensive testing of the OAI feed/configuration! thanks

DiegoPino commented 3 years ago

@dmer will be ready by next week. Work has started and it is easier than expected. More to come, will also support OAI-ORE

dmer commented 2 years ago

hello! I'm working my way through some expected requirements as we try to start fitting folks' requirements onto the reality of the library internet in which OAI will be a legacy component for a while still. No time pressure on this, but I just wanted to volunteer to do testing on an OAI endpoint. I have had (sadly) a lot of experience getting the necessary results from that admittedly old and cranky "API" with its verbs and whatnot . If this is already in the RC3 latest that I'm running on my test server, I just need a point in the right direction for how to access it? I see the simple DC twig is OAI_DC ready. I'm just not clear on how to structure URL - something like the expected /oai2?verb=ListSets thanks!

patdunlavey commented 1 year ago

Hi @DiegoPino . Following up on your suggestion about building OAI for archipelago starting with the rest_oai_pmh module, and adding a metadata mapping module along the lines of the rest_oai_pmh_jpcoar example, I have it working. It's actually really sweet, and I'd like to propose that I provide it back as a submodule of format_strawberryfield. I'll make a PR and we can take it from there!

DiegoPino commented 1 year ago

Yep. Sounds good! Please do

Thanks