Creating resources derived from the knowledge graph using Event Sourcing and CQRS patterns

[pietercolpaert]

Pitch

An authoritative write resource can have multiple Knowledge Graphs views on top of it. Describing -- and executing -- workflows on top of these authoritative resources creating such derived views (cfr. the architectural event sourcing pattern or CQRS pattern) can be the glue between seeing a Solid pod as a file-based system vs. a knowledge graph.

Examples of Knowledge Graph resources:

• Querying views [1] such as
  • TREE Collection views for autocompletion, geospatial search, or time-based search
  • a TPF interface for graph-based querying
  • a SPARQL endpoint
  • a GraphQL API
• Inferred resources such as
  • The results of a function (e.g., geocoding your location: if your current long/lat is X,Y, then your current city is A - you want this processing to happen on the server, as the original long-lat pair should not be shared with any third parties)
  • The results of a query (e.g., all people you’ve written a paper with together in a resource co-authors)
  • The results of a reasoning process (e.g., all people you know based on multiple relations and logic)
• Shape translations or mappings:
  • If the write-location works through a non-RDF resource, a mapped resource can contain the Linked Data
  • If the write resource uses schema.org, a shape translated resource could contain the FOAF translation
    • Concatenations of resources: a concatenated resource of multiple other write-resources (e.g., concatenating all resources in a chat-container in order to speed up authorization)

[1] @RubenVerborgh’s reflections of knowledge contains the general idea. This solution should provide one way to glue together a knowledge graph with the resource-based views on top of Solid pods.

Desired solution

Acceptance criteria

• The creator of a knowledge graph resource needs to have the right permissions to create a derived resource. Security experts need to work out a security approach to ensure data like location history cannot be leaked.
• We identified the changes needed in the Solid specifications in order to support these kind of data workflows: propagating changes in an atomic fashion
• There’s a prototype in which a multi-step workflow in the CSS can be configured and published. The prototype could work with external orchestrators, or work from within the CSS if the functionality is enabled. Multi-step means that you first write to a structure, then create a view, and then read from the view the data you need for your application
• We have a strategy for the Solid ecosystem on what the minimum functionality is a Solid server must provide vs. all processing capabilities that could be extensions, and how consumers deal with such heterogeneous capabilities of servers.

Pointers

• Related Solid spec issue: https://github.com/solid/specification/issues/390
• Related issue in challenges: #17
• Architecture patterns: Command and Query Responsibility Segregation (CQRS) pattern and Event Sourcing pattern
• Vocabularies:
  • FnO: https://fno.io
  • P-Plan: https://purl.org/net/p-plan

Scenarios

See the use cases on chat #1 and location history #10

[pietercolpaert]

Through the concept of streams and processors, we’re working on a way to describe derived resources in the SDS specification: https://w3id.org/sds/specification

[pheyvaer]

@RubenVerborgh What changes do you expect here?

[RubenVerborgh]

A challenge is a technical problem applied to a concrete use case with as output a demo. I'm missing the use case and demo parts.

[pietercolpaert]

Could we maybe add a separate type for an architecture proposal that could be applied in multiple challenges?

I could add a use case and a demo, but then it would basically be #10, but a bit more generically described

[RubenVerborgh]

I think it's fine to have #10 actually, or to suggest this as a solution to #10.

One reason we're asking for a specific use case and a demo, is that those are tasks that can be finished. The technical challenge described above is fundamentally never finished: one can always find new ways to derive resources.

So can you think of the a case that demonstrates the kind of complexity you at least want? I'd suggest to keep the text above as-is, and to create a new issue. (We could later indeed decide to collect architectural proposals.)

[pheyvaer]

@pietercolpaert Did you have the chance to look into making the necessary changes?

[pietercolpaert]

I still wonder whether we’d rather call this a design issue then and document it elsewhere. For me #10 remains the concrete demo there.

[pheyvaer]

@RubenVerborgh What should we do with this? You said to keep the text, but does that mean that no changes are needed anymore?