SEMICeu / LinkedDataEventStreams

The Linked Data Event Streams specification
https://tree.linkeddatafragments.org/linked-data-event-streams/
23 stars 9 forks source link

Standardize client iterator #31

Open pietercolpaert opened 1 year ago

pietercolpaert commented 1 year ago

While the design of some LDES-es can be really simple, just keeping a history log of all pages and members processed to keep the state will not scale to really big LDES-es.

In earlier work, we have introduced the ldes:timestampPath to point at a property that can be used to keep the state in a different way: just resume from that timestamp.

If would be good to have a standard algorithm described in the spec to explain how a client must pause and resume from a certain state.

Coined by @sandervd

sandervd commented 1 year ago

To make this work efficiently, I think we would need a way to communicate to the client that the view uses a unidirectional linked list as pagination model between fragments. If this is the case, the client can keep the last consumed fragment uri and the timestamp of the last processed member as state, and can continue from there on?

pietercolpaert commented 1 year ago

Not convinced a unidirectional fragmentation is needed: the relations however do have to use the ldes:timestampPath as tree:path.

sandervd commented 1 year ago

If we go with this idea of 'profiles', that allow the client to discover how a view can be consumed, we could define the minimal relations that should be present in said profile? e.g. tree:GreaterThanRelation is required in each fragment? Having someting with even more simplistic semantics like nextOrderOfArrivalFragment would be nice to have, as in theory a fragment could contain multiple GreaterThanRelation relations? If we want to keep the itterator logic as simple as possible, being able to have a explicit order beween fragments becomes important.