Shape / policy sets, union and intersection

[Dexagod]

A Linked Data Notification processing agent should be able to evaluate multiple different filters over the inbox. Similar to how in a mail inbox, multiple processing rules can be set to rout incoming emails.

In the current implementation of the Linked Data Notifications agent the agent only filters the messages that are validated by all passed shape filters (union). This is insufficient for more elaborate use cases. E.g. the orchestrator component for Mellon requires the evaluation of multiple policies, that may all have a different activation condition (shape).

Ideally, the interface of the agent should enable clients to decide exactly which shapes should be matched by emitted notifications. This could be made possible by enabling a client to specify a union and intersection operation.

Additionally, a client may be interested to retrieve different sets of notifications, matching different sets of filters. For this, we could allow multiple sets of filters to be created and passed. This enables the agent to return iterators for all different filter sets containing the matching notifications.

An example for the config file / separate filter file including both ideas:

{ 
  ... 
  "ex:filters": 
  [
    {
        "ex:name": "filterset1"
        "ex:intersection":
        [
            "ex:union": 
            [
                { 
                     "@id": https://site.org/shape1.ttl
                     "@type": ex:shapeFile
                }, 
                { 
                     "@id": file:///home/alice/shapes/shape2.ttl 
                     "@type": ex:shapeFile
                }
            ], 
            { 
                 "@id": https://site.org/shape2.ttl
                 "@type": ex:shapeFile
            }
        ]
    }, 
    { 
        "ex:name": "filterset2"
        ...
    }, 

  ],
  ...
}

An example for the return value of the agent:

// The filters are contained in config.
// A separate config with additional information could be added as a watchNotifications() parameter.
const agent = new Agent(config)
const resultsMap = await agent.watchNotifications()
const filterSet1Results : AsyncIterator = resultsMap['filterset1'];
const filterSet2Results : AsyncIterator = resultsMap['filterset2'];

// In the current implementation, only an iterator of the matching notifications is returned.

[Dexagod]

In the case of the filter sets, how should we then handle the default case where no filters are passed. maybe through a default keyword? e.g.

const resultsMap = await agent.watchNotifications()
const iterator : AsyncIterator = resultsMap['default'];

Or is this not intuitive enough

[rubensworks]

@Dexagod Are we sure that this union/intersection capability doesn't exist yet in SHACL/ShEx? If it does, then it might be easier and more portable to just use that.

[Dexagod]

SHACL indeed has union / intersection support, my bad. So these can directly be incorporated in the shape.

The reason for the separate shapes and separate returns is because policies for e,g, inbox processing consist of a match clause (in this case a shape to match), and an action. However I do not see a way to semantically describe all actions that one might require, so it makes sense to return all notifications matching a given shape, and have the developer implement the relevant actions himself.