Orchestrator (autonomous agent) to forward LDN Notifications to the correct LDN Inbox based on N3 rules

[phochste]

Pitch

A Maintainer of a (Solid) LDN Inbox would like to see some automated processes happening when particular LDN Notification messages appear in one of her Inboxes or are being sent by the Maintainer. Possible automated processes include:

• Auto reply to some messages
• Forward the notification to the LDN Inbox of another autonomous agent
• Append outgoing messages to an Outbox/Sent container
• Generate a new Notification to be sent to a particular LDN Inbox

What should happen after receiving an incoming our outgoing message is described by an N3 rule book (rules). These rule books could require read access to the Pod (e.g. to check the existence of a resource on the Pod). E.g. a rule (using natural language just for the example):

If the Notification is about a Resource, AND Resource exists on the Pod , THEN generate new Notification to XYZ

The automated processes and rules are executed by an autonomous agent, the Orchestrator, on behalf of the Maintainer.

Desired solution

The desired orchestrator solution as the following pre-conditions:

• Read/Write access to the Maintainer LDN Inbox
• Read/Write access to the Maintainer LDN "Outbox/Sent" box (to which outgoing notifications should be sent by the maintainer)
• Read access to one or more Rule books (resources that contain the rules) : currently these are expressed as Notation3
• Knowledge of the web locations of all these resources
• Well-formed messages are ActivityStream 2.0 notifications

What the orchestrator should do:

• Have a WebId (to get read/write access)
• Watch the Maintainer LDN Inbox and Maintainer LDN "Sent" box for incoming messages
• Validation of messages (only some shapes are allowed)
• Deduplication of messages (based the notification identifier)
• Optional: signature checks
• Feed the Notification to a rule engine (currently N3 engines such as the eye reasoner are in focus)
  • The reasoner should produce zero or more of
    • A Notification message that should be send to a particular inbox
    • A Notification message that should be appended to a particular container
• An implementation of:
  • A notification sender (to a particular inbox)
  • A notification appender (to a particular container)

Extra concerns:

• The orchestrator should keep a provenance trail of actions
• The orchestrator should provide an (web) API to:
  • Create an orchestrator for a particular combination of Inbox + Sent box + container + rulebook
  • Start | Stop orchestrator
  • Destroy orchestrator
• It should be possible to reuse the components (Inbox Watcher, Validator, Deduplicator, Rule Engine, Notification Sender, Appender) with own implementations.

Acceptance criteria

A minimal demonstrator should:

• Be an autonomous server process
• That watches an LDN Inbox for which it has read-only access
• Has access to a Notation 3 rule at https://somepod.orf/alice/my-autoreply.n3

# Auto reply Hello world 
PREFIX as:   <https://www.w3.org/ns/activitystreams#>
PREFIX pol:  <https://www.example.org/ns/policy#>
PREFIX fno:  <https://w3id.org/function/ontology#>
{
    ?id a as:Announce .
    ?id as:actor ?source .
    ?id as:target ?target .
}
=> 
{
  [ pol:policy [
      a fno:Execution ;
      fno:executes ex:sendTarget ;
      ex:notification [
            a as:Announce ;
            as:inReplyTo ?id ;
            as:actor ?target ;
            as:target ?source;
            as:object    [
              a as:Note ;
              as:content "Hello world" 
            ]
      ]
    ]
  ].
}.

• Executes the rule above for all incomming messages
• When the result of rule processing produces an fno:Execution it executes an implementation of sendTarget which send the notification to the Agent described by as:target.

Pointers

https://mellonscholarlycommunication.github.io/spec-orchestrator/

Scenarios

Typical use cases:

• When a researcher finishes a publication, automatically send a notification to configures parties (notify friends, library catalog, department website)
  • A Researcher app just sends one "Announce" notification to the orchestrator, the orchestrator has a configuration of rules who to forward the message
• When a research finish a publication, automatically update her public log book of public announced publications
  • The orchestrator has a rule to add an Event to a Container when a new "Announce" from the researcher is read.
• A researcher finished a publication and sends a notification to the orchestrator
  • The orchestrator provides a list of venues based on some rules (e.g. the researcher dowloaded a conference season rule book) back as a notification to the researcher
• A conference accepts a publication of a researcher and send a "Announce accept" notification to the researcher's LDN Inbox.
  • The orchestrator has a plugin to automatically update the metadata of the publication stating that the publication has been accepted
  • The maintainer has a rule to execute the update metadata when receiving a notification from the conference.
• A journal published a publication of the researcher and send an "Announce published" notification to the researcher's LDN Inbox.
  • The maintainer has a rule to execute the "add back link" plugin when such a notification is received.
• Every 15 March a LDN Notification should be send to the researcher reminding her she needs to update and check her university bibliography

[phochste]

Related to https://github.com/SolidLabResearch/Challenges/issues/47

[phochste]

Related to https://github.com/SolidLabResearch/Challenges/issues/35

[RubenVerborgh]

Do we have a concrete use case to apply this one to?

[phochste]

Added some

[phochste]

Work in progress in Koreografeye project

[pheyvaer]

@phochste Does this mean that you are actively working on this challenge?

[phochste]

Yes

[github-actions[bot]]

Please provide a status update about this challenge. Every ongoing challenge needs at least one status update every 2 weeks. Thanks!

[phochste]

@pheyvaer The Koreografeye is ready to be evaluated. It provides the functionalities of the acceptance criteria. What are the required steps to get into the new status?

[pheyvaer]

@phochste The instructions are here. Can you also provide a link to the repo with the code?

[phochste]

https://github.com/eyereasoner/Koreografeye

[github-actions[bot]]

Please provide a status update about this challenge. Every ongoing challenge needs at least one status update every 2 weeks. Thanks!

[phochste]

Two examples repostories are added for Koreografeye how to use it:

• KoreografeyeDemo - how to do orchestration
• KoreografeyePluginDemo - how to create a new side-effects plugin

[phochste]

All plugins have now a separate project:

https://github.com/eyereasoner/Koreografeye-Plugin

The documentation of Koreografeye was simplified to explain the essentials to get up and running:

https://github.com/eyereasoner/Koreografeye

[pheyvaer]

Nice! For the challenge specifically, will you use a plugin in the demo that has something concrete instead of "some debugging output"?

[pheyvaer]

@phochste Could you answer these two questions for the report? Thanks!

• What were important technological decisions and assumptions you made?
• What are the lessons learned about your experience as a Solid developer?

[phochste]

@pheyvaer

What were important technological decisions and assumptions you made?

The solution should be expected to be part of a larger existing framework. Koreografeye doesn't implement some features on purpose (like scheduling, rate limits, input and outputs, priorities etc). For these solutions there are already existing tools that can provide that. Koreografeye should be able to be just one added component to for instance an Apache Nifi installation.

Composition of Koreografeye parts such as input , orchestration an policy execution should be possible via the command line. The JavaScript API is just an added feature not the core of the design.

A lot of plugins could be created for policy enforcement and reasoner implementations. Component.JS was used to facilitate this.

Koreografeye makes a strict separation between reasoning and policy execution. We don't advise to create any side-effects in the reasoning part for two reasons:

• The rule book that the reasoner gets as input can be from external sources. We don't want arbitrary execution of code.
• During the reasoning phase there is little control or no over possible side-effects (when they are executed, in which order and how often).

The assumption that were made:

• Retrieving input from a (Solid) source is a solved issue. The Koreografeye starts when an RDF resource is copied into a local storage location accessible for Koreografeye
• Retrieving the rule book from an external source is a solved issue. Koreografeye expects a local storage location with zero or more rule book N3 or N3S (rdf surfaces) files.
• The order in which policy enforcements are executed are not relevant.
• Rule books don't interfere .. each rule book gets a new copy of the RDF input data

What are the lessons learned about your experience as a Solid developer?

• Solid application development is not only about web programming.
• Logic and reasoning can be external to your implementation.
• It is possible to provide very limited remote execution near or inside a Solid pod when logic and handling side-effects can be separated.
• Although with Bashlib we have a command line tool to access a Solid pod and in this way can manage content, remote server side Solid data management is in general not a solved issue. Bashlib requires assumptions about specialised authentication methods for each Solid implementation.

[pheyvaer]

Thanks! For the lessons learned can you explain more?

Solid application development is not only about web programming.

What else is in included?

Logic and reasoning can be external to your implementation.

And what is good/bad about this?

It is possible to provide very limited remote execution near or inside a Solid pod when logic and handling side-effects can be separated.

Is this good/bad? This reads more as a conclusion to the work itself. Unless I misunderstand what you mean.

remote server side Solid data management is in general not a solved issue.

What is missing? What is not solved?

[pheyvaer]

Reminder @phochste

[phochste]

Solid application development is not only about web programming.

What else is in included?

For real Solid integration, processes need to be automated. Resources need to be made available, updated, delete also when no user is available, no web browser is opened. In a decentralized world, we can't expect that a Solid Pod will keep on adding logic to provide this automation steps. External services will provide this automation capacity.

Logic and reasoning can be external to your implementation.

And what is good/bad about this?

The good part of having logic external to your application is that programmers only need to worry about their own problem domain. E.g I want that linked data notifications that are send to my Pod are forwarded to my email address. This logic shouldn't be implemented by every Solid application in the work that send notifications to my Pod. What do they care what kind of extra business rules are required when an notification arrives in the LDN Inbox?

The bad part is like everything else in a decentralized world. When the logic is spread around many services:

• who keeps an overview of what process flows are running on your data.
• who keeps an eye on resource dependencies (e.g. can I delete/update a resource when I don't know if another logic still needs it)

It is possible to provide very limited remote execution near or inside a Solid pod when logic and handling side-effects can be separated.

Is this good/bad? This reads more as a conclusion to the work itself. Unless I misunderstand what you mean.

An orchestrator could in principle also run inside a pod. But most probably one wants to provide it very limited capabilities to prevent users to run arbitrary code on the pod server itself. E.g. an orchestrator that works only as an LDN Inbox rule engine like one has in Outlook or Thunderbird would allow only moving incoming notifications to a specific container.

remote server side Solid data management is in general not a solved issue.

What is missing? What is not solved?

We are currently depending on an authentication mechanism that only works for CSS (tokens). Every Pod implementation has its own mechanism to provide access tokens without needing to got a Web client every X minutes/hours/days.

[pheyvaer]

You find the report here.