Design for plugin-based Elasticsearch Transformations

samuel-kerrien commented 3 years ago

Follow up of this ticket: Transformation before ElasticSearch indexing #2549

Acceptance Criteria

validate with users (@kaij and @jonathanlurie)
Document describing the proposed solution
- The solution should accommodate the two usecases provided by @kaij
Meeting to review with the team
Meeting to review with @bogdanromanx

Meeting to review with @kaij

imsdu commented 2 years ago

Draft PR that gives some insights of what should be done here: https://github.com/BlueBrain/nexus/pull/2886

NB: We would rather use the term Pipe instead of Transformation

Property of pipes:

They are identified by an unique name
They can be chained and form a pipeline
They can be used to transform or to filter out a resource before its indexing
Some pipes will require a Context depending on the pipe (ex: to define the types to filter out, a construct query to transform the resource, ...)

Internal changes in Delta

All existing operations (resourceSchemas, resourceTypes, sourceAsText, includeMetadata, includeDeprecated, ...) will be rewritten as pipes
A Pipe allowing to apply a construct query to transform the resource will be added
A migration will be run on Delta to migrate ElasticsearchViews to their new model
The indexing process will be updated
Delta won't start if two pipes share the same id

Changes in the ES views endpoint

Creates/updates

A new property will appear in the payload when creating/updating

Example:

{
  ...,
  "pipeline": {
    {
     "name": "excludeMetadata"
    },
    {
      "name": "filterTypes",
      "description": "We wish only to keep persons here",
      "context": {
        "types": ["schema:Person"]
      }
    },
    {
      "name": "[pipeName]",
      "description": "[Optional description]",
      "context": {
        "property": "[Optional context]"
      }
    }
  },
  ...
}

When defining the pipeline property, none of the legacy resourceSchemas, resourceTypes, sourceAsText, includeMetadata, includeDeprecated properties can be defined
When a pipeId is not found or if the provided context is invalid, a rejection is returned.

Reads

For retro-compatibility purposes when getting a view:

All legacy properties will be computed from the pipeline
The pipeline will always be present

Documentation

Documentation for Elasticsearch views will be updated according to the changes
A new page describing the builtin pipes will be added.
A new how-to describing on how to create a Pipe

Checking on views

After a new Delta version / new version of a Pipe / removal of a Pipe, a pipeline can become invalid. So indexing should not be started for them and these views should be advertised (through an endpoint ?) so that users can be alerted and can fix them.

Testing a pipeline

Create an endpoint where people can test the pipeline against a bunch of resources with actually creating it / indexing

Extend Pipes to other types of views

Generalize Pipes to Blazegraph and Composite Views

Plan to redesign views endpoints

Break retro-compatibility to make endpoints more understandable/maintainable both on user and developer point of view

Tasks

1.7:

Update Elasticsearch view model
Migrate Elasticsearch views to new model (blocked by 1.)
Update Elasticsearch indexation to integrate the pipeline (blocked by 1.)
Update and keep Elasticsearch endpoints retro-compatible to handle pipelines (blocked by 1.)
Documentation

After 1.7:

Add an endpoint to detect broken views
Add an endpoint to test views and pipelines
Extend pipelines to other views
Redesign view endpoints

samuel-kerrien commented 2 years ago

I see that you are considering adding an endpoint to discover broken views, would it not be simpler to describe the status of a view when listing them (ref: https://bluebrainnexus.io/docs/delta/api/views/index.html#list-views) ?

samuel-kerrien commented 2 years ago

Are to able to fit @kaij's usecase into this proposal ?

imsdu commented 2 years ago

I was thinking of another endpoint which would act as an healthcheck with the failing views and the reason (retro-compatibility has been broken, the pipe does not exist anymore, ...).

An endpoint that could be used by a probe that could trigger an alert if at least a view is failing.

I am sharing it with the team for now to get some validation between us first. For example, it is very Delta-focused for now, there should be some impacts on Fusion to discuss too.

BlueBrain / nexus