Add dates to PASS model

[birkland]

Dates (created, last modified) are present in Fedora, but are not universally part of the PASS model, and therefore cannot be queried by Elasticsearch. Figure out how to make created and last modified dates queriable, so that it's possible to ask for "Funders added in the last month", for example.

Figure out if it makes sense to merely add dates to the index, or if the PASS model itself needs to be updated (making dates first class citizens in the model).

Some considerations and nuances:

• Directly using Fedora's created and last modified dates may be tricky, as they are "server managed triples", and Fedora will not accept user-provided values for server-managed triples by default.
• Requiring applications to provide dates in PUT/POST requests would be a fairly radical change in model usage
• We may or may not want applications to be allowed to PUT/POST their own date values in he first place.

[markpatton]

Some background:

• Elasticsearch supports a date field type in documents and date fields can be the subject of range queries.
• The ember-fedora-adapter supports Ember date fields and maps them to xsd:dateTime in JSON-LD.

Options:

• Add create/modify date attributes to data model and JSON-LD serialization.
  • Attributes could be populated by Fedora date properties with some magic on the server
  • Attributes would have to be ignored by server on write or ember-fedora-adapter updated to support read-only object fields
• Manually add create/modify date attributes to Elasticsearch documents only
  • This requires modifying pass-indexer to grab the attributes from Fedora objects and mangle the JSON-LD a little more.
  • A consequence is that these attributes are invisible to Ember. (Technically they could be seen on search, but then a mismatch between what is retrieved from Fedora and what is retrieved by search could be a problem.)
  • The data model becomes more complicated, because now we have Elasticsearch only versions of objects.
• Have Elasticsearch automatically add create/modify attributes to documents.
  • Create an Elasticsearch script which sets create/modify date fields on an object.
  • Modify pass-indexer so that document updates become a scripted_upsert using that script.
  • See this example.
  • A consequence is that these dates will not match the Fedora dates and have different semantics. They will also change on a full reindex of Fedora.
  • There is the same consequence to Ember as above.

[birkland]

@markpatton Would you be able to weigh the pros and cons of the various approaches, and provide a recommendation?

[markpatton]

@birkland Will do.

[emetsger]

Figure out if it makes sense to merely add dates to the index, or if the PASS model itself needs to be updated (making dates first class citizens in the model).

I will add that to me it doesn't make sense to not add dates to the formal PASS model.

Querying ES say, for the most recent 10 submissions, and not exposing their dates in the model doesn't allow me to sort on, store, or range over the date locally using the PASS model. The results may be iterated by the caller in document order (i.e. the order that is returned by ES), but I can't manipulate the order or reason about the temporal range represented in the results.

Ideally dates are not required to be set by the caller when creating or updating objects (i.e. they're auto-magic, populated on the server side), but if they are set by the caller, they're honored (not overwritten server-side). Obviously this falls on one end of the continuum of options, but if we're going to do it, let's do it.

[markpatton]

Review of options with recommendation: https://docs.google.com/document/d/1lliGYBk4Luz2syNZHH1ScESCZHKtEmSjOcteayppiaQ/edit?usp=sharing

[birkland]

One of the complications here is that PASS doesn't really have a well-defined API. When the PASS data model proves insufficient, we can peer into the implementation details and look at Fedora's raw RDF. We can use Fedora's "inbound relationships" feature to effectively perform some kinds of queries that are difficult or impossible in elasticsearch, etc.

With that context, the proposed solution of indexing Fedora's date fields effectively states that there probably ought to be read-only "system" date attributes in PASS's API that adopt the same semantics as Fedora's created and modified dates, and are entirely managed by the system. By not formally updating the model, indexing these fields ends up being a de facto partial implementation of read-only "system" dates as far as PASS's "API" is concerned.

If we think of the door being open to completing this work some day as part of forming a well-defined PASS API, then we can imaging possibly deciding to implement option (1) some day, and add these date properties to the PASS model for real. In that light, we probably ought to view (2) as having the potential to lead to (1) down the road, and design the field/representation/doc accordingly. Since the proposed date fields are read-only "system" fields, I think we'd need some way to set them apart.

@markpatton Do you agree with this perspective, and the need to somehow distinguish these "system" fields from the others? If so, can you propose (maybe update the doc) a specific way of doing that? Off the top of my head, I can think of

• prefixing fields in the JSON doc with system:, i.e. system:createdDate
• Having a system object in the JSON, e.g.

  {
  "foo": "bar",
  "system": {
  "createdDate": "2018-05-10T17:13:44.352Z"
  }
  }
  

The challenge: While indexed docs aren't formally JSON-LD, the PASS model representation is. So whatever we do would need to make sense as JSON-LD.

[markpatton]

@birkland I'm not sure they really need to be distinguished in the JSON-LD representation. Documentation and appropriate errors from APIs may be enough. Having a separate system object is a significant increase in complexity for clients, but certainly could be done. If we want to distinguish system properties in the JSON-LD, another option is to have a different namespace for them. Then depending on how we define the context we could have our canonical representation be system:createDate or createDate etc.