Services API Design

[daniel-dgi]

In preparation for the next community sprint, we'll be overhauling the middleware services to utilize PHP. While we're at it, we're reconsidering the API design in general, and are attempting to move away from Drupal specific inputs to standardizing on RDF generated from Drupal.

Please use this issue to discuss what features we really want out of services (Fedora API extensions, PCDM CRUD Operations, etc...), and what we think the interface should look like (e.g. route structure, required and optional arguments, etc...).

[ruebot]

Some what related, do we need to bring porkpie over to this GitHub org?

[ruebot]

w/r/t chullo, are there any more REST methods we need to setup? Or, do we move straight into the tunnel of abstraction? :smile:

[daniel-dgi]

Here's a first pass that should get us up to the point we're at now, but without accepting anything Drupal specific. Comments/critiques welcomed.

Transaction Service

Used to start, commit, and rollback transactions.

Endpoint: http://localhost:8080/islandora/transaction/

Actions:

• POST http://localhost:8080/islandora-services/transaction/
  • Creates a transaction in fedora, returning the id of the newly made transaction
• POST http://localhost:8080/islandora-services/transaction/{tx_id}/commit
  • Commits a transaction
  • Returns 204 - No Content
• POST http://localhost:8080/islandora-services/transaction/{tx_id}/rollback
  • Rolls back a transaction
  • Returns 204 - No Content

Resource Service

Exposes basic CRUD operations on repository resources identified by a UUID.

Endpoint: http://localhost:8080/islandora/resource/

Actions:

• GET http://localhost:8080/islandora/resource/{uuid}?tx={tx_id}
  • Returns RDF metadata for the resouce identified by the provided UUID. Respects all headers the Fedora 4 API respects.
  • Optional transaction id will ensure the returned the RDF represents the current status of the resource within said transaction.
• POST http://localhost:8080/islandora/resource/?tx={tx_id}
  • Creates a resource in Fedora 4. Respects all headers the Fedora 4 API respects.
  • Optional transaction id will ensure the resource is created within said transaction.
• PUT http://localhost:8080/islandora/resource/{uuid}?tx={tx_id}
  • Updates a resource in Fedora 4. Respects all headers the Fedora 4 API respects.
  • Optional transaction id will ensure the resource is updated within said transaction.
• PATCH http://localhost:8080/islandora/resource/{uuid}?tx={tx_id}
  • Applies a SPARQL/Update query against a resource in Fedora 4. Respects all headers the Fedora 4 API respects.
  • Optional transaction id will ensure the resource is updated within said transaction.
• DELETE http://localhost:8080/islandora-services/resource/{uuid}?tx={tx_id}
  • Deletes the resource in Fedora 4.
  • Optional transaction id will ensure the resource is deleted within said transaction.

Collection Service

Convienence operations for pcdm:Collections in Fedora 4.

Endpoint: http://localhost:8080/islandora/collection/

Actions:

• POST http://localhost:8080/islandora/collection/?tx={tx_id}
  • Creates a pcdm:Collection in Fedora 4. Respects all headers the Fedora 4 API respects. Adds the appropriate indirect containers to manage the pcdm:hasMember relationship and its inverse.
  • Optional transaction id will ensure the resources are created within said transaction.

Object Service

Convienence operations for pcdm:Objects in Fedora 4.

Endpoint: http://localhost:8080/islandora/object/

Actions:

• POST http://localhost:8080/islandora/object/?tx={tx_id}
  • Creates a pcdm:Object in Fedora 4. Respects all headers the Fedora 4 API respects. Adds the appropriate containers to manage the pcdm:hasMember and pcdm:hasFile relationships and their inverses.
  • Optional transaction id will ensure the resources are created within said transaction.

Membership Service

Operations to add/remove members from pcdm:Objects and pcdm:Collections.

Endpoints:

http://localhost:8080/islandora/object/{uuid}/members

http://localhost:8080/islandora/collection/{uuid}/members

Actions:

For brevity, only one of the two endponits is described in the following section.

• 'GET' http://localhost:8080/islandora/object/{parent_uuid}/members?tx={tx_id}
  • Retrieves a list of members associated with object identified by parent_uuid.
  • Optional transaction id will return the state of the list of members within said transaction.
• POST http://localhost:8080/islandora/object/{parent_uuid}/members/{child_uuid}?tx={tx_id}
  • Adds the resource identified by child_uuid to the object/collection identified by parent_uuid
  • Optional transaction id will ensure the operation is performed within said transaction.
• DELETE http://localhost:8080/islandora/object/{parent_uuid}/members/{child_uuid}?tx={tx_id}
  • Removes the resource identified by child_uuid from the object/collection identified by parent_uuid
  • Optional transaction id will ensure the operation is performed within said transaction.

File Service

Lists all files for a pcdm:Object.

Endpoint: http://localhost:8080/islandora/object/{uuid}/files

Actions:

• 'GET' http://localhost:8080/islandora/object/{parent_uuid}/files?tx={tx_id}
  • Retrieves a list of files associated with object identified by parent_uuid.
  • Optional transaction id will return the state of the list of files within said transaction.

Thumbnail Service

CRUD operations for thumbnails

Endpoint: http://localhost:8080/islandora/object/{uuid}/thumbnail

Actions:

• 'GET' http://localhost:8080/islandora/object/{uuid}/thumbnail?tx={tx_id}
  • Retrieves the thumbnail for the object identified by the provided uuid. Setting the accept header to an RDF mimetype will return the RDF for the file. Setting it to a binary mimetype will return the contents of the file.
  • Optional transaction id will return the thumbnail within said transaction.
• 'PUT' http://localhost:8080/islandora/object/{uuid}/thumbnail?tx={tx_id}
  • Saves the thumbnail provided in the message body to the object identified by the provided uuid.
  • Optional transaction id will save the thumbnail within said transaction.
• 'DELETE' http://localhost:8080/islandora/object/{uuid}/thumbnail?tx={tx_id}
  • Re,pves the thumbnail for the object identified by the provided uuid.
  • Optional transaction id will delete the thumbnail within said transaction.

Preservation Master Service

CRUD operations for preservation masters

Endpoint: http://localhost:8080/islandora/object/{uuid}/preservationMaster

Actions:

• 'GET' http://localhost:8080/islandora/object/{uuid}/preservationMaster?tx={tx_id}
  • Retrieves the preservation master for the object identified by the provided uuid. Setting the accept header to an RDF mimetype will return the RDF for the file. Setting it to a binary mimetype will return the contents of the file.
  • Optional transaction id will return the file within said transaction.
• 'PUT' http://localhost:8080/islandora/object/{uuid}/preservationMaster?tx={tx_id}
  • Saves the preservation master provided in the message body to the object identified by the provided uuid.
  • Optional transaction id will save the file within said transaction.
• 'DELETE' http://localhost:8080/islandora/object/{uuid}/preservationMaster?tx={tx_id}
  • Removes the preservation master for the object identified by the provided uuid.
  • Optional transaction id will delete the file within said transaction.

[daniel-dgi]

@ruebot We'll start out by abstracting a Fedora response and having Chullo work at that sort of 'raw' level from Fedora. Then these individual services will provide utility functions that resemble what Chullo does right now. Those utility functions will then be exposed in a micro-framework actually implement the REST service.

We'll be essentially building a new porkpie from the ground up. It may end up being too much for a single library. I'm playing it by ear until we can implement some stuff. Then we'll pull it out into a separate library if there appears to be a lot of commonality.

Let's talk about the API first, though :D

[ruebot]

@daniel-dgi Excellent. Thanks for this! I'm understanding things a lot better now.

...my only critique/feedback for now, would be that we should probably just rename some of the services base on the PCDM use extension:

• preservationMaster → PreservationMasterFile
• thumbnail → ThumbnailImage

...might be splitting hairs. But consistency is good, right :smiley:

[DiegoPino]

@daniel-dgi and @ruebot, i think it's great. Will triple store operations be handled also as a service? Also, Membership Service could be convert this to Semantic Service or something similar? This way we could handle other relations, not necessary to fix this to pcdm? But still keep the notion of object to URI (or IRI) relation as input. I know PCDM is THE Ontology, but i wish we could get some extra flexibility on that side.

Thanks!

[whikloj]

@DiegoPino I am assuming that the Membership Service assumes a pcdm:hasMember predicate, perhaps this could implement a more abstract Semantic Service so you could use that service for use with alternate ontologies?

Also, @daniel-dgi I saw you put in return code for the Transaction Service but:

1. no error codes
2. no return/error code for any other services

Is that an oversight or perhaps too early to worry about.

[daniel-dgi]

@whikloj I haven't thought them through all the way. Just throwing this out there to get the conversation started. We'll need to work through detailed response scenarios together.

Most of them just mirror what the Fedora 4 API returns, but we are on the hook for the membership, file, and derivative services.

[daniel-dgi]

@ruebot we could also just go back to our old datastream names (OBJ, TN) etc... I wasn't sure what to put. To be honest, i'm not happy with even 'preservationMaster' because I think it's tool long and i don't like camel-case (or capital letters, period) within urls.

[daniel-dgi]

@DiegoPino figured we'd just expose the triplestore endpoint directly in a read only fashion. writes and other things should happen behind the scenes in the service or asynchronously in response to a message.

[daniel-dgi]

FYI, Google doc with all this is here

[DiegoPino]

:+1:

[ruebot]

@DiegoPino @whikloj should we bring that Google Doc into the the Project Plan doc? Make it an appendix?

[DiegoPino]

@ruebot, yeah, that is a good idea. It's somehow lost in the forum right now.

[whikloj]

👍

[tpendragon]

So Hydra's starting to think about what a middleware layer API for PCDM models would look like, so I wanted to ask a couple questions:

1) This looks like it's a proxy over LDP with a couple magic containers for building objects/collections - why the decision to go there and build up HTTP transactions rather than defining an API that can support nested creates which your backend can handle as a transaction to Fedora, if necessary?

2) How often do you intend to fire off these requests? It seems to me that Hydra and CLAW are in two very different places - if you just never did any of this you'd have a repository with all your data and images, yeah? Is the goal to keep Drupal and Fedora constantly in sync, or do you see Fedora as a (maybe with different wording) "preservation store," that contains all your RDF assertions as well as the model and files attached to them?

[whikloj]

Hi @tpendragon, As a developer who never got his CompSci, I sometimes lack the understanding of some terms. So if I am off in left field, feel free to reign me in.

That being said:

1. The services designed in this Github issue are more comprehensive than the actual implementation seems to be working towards. A lot of our code is simply wrappers around the Fedora API and a more complex service for the unique PCDM structure of things which use the lower level microservices.

   When you say "why ... build up HTTP transactions rather than defining an API that can support nested creates...", I'm not sure I fully understand the distinction.

   But I would say, that we wanted to leave the "how" to build the resource to the front-end. So if we design a new "content model" in Drupal it could simply call these middleware services as it needs to create the new structure. We are trying hard to avoid putting all the structure into the middleware code.

   This allows us a couple of flexible options to create PCDM resources:

   1. You can use the middleware with a different front-end.
   2. You can generate new PCDM structures without altering the CLAW middleware.

2. Again, I might have missed what you meant by "if you never did any of this you'd have..."

   We are attempting to allow for either a loose sync between Fedora and Drupal (my personal desire) where your masters and their metadata are available in Fedora, but derivatives and anything generated from the masters are not. In this case certain updates on the Drupal side will sync to Fedora, but all updates to the Fedora side need to be synced back to Drupal.

   But there are community members whom have expressed the desire to store all content in both Fedora and Drupal (in fact I believe we have heard suggestion of trying to make Fedora a backend for Drupal). So this means all changes from either side need to be synced.

All of this being said, we are now 10 days from having a tech lead (👏 ). So...he (or @DiegoPino) will probably be much better able to answer your questions.