real-time replayer doesn't replay as a real-time stream compliant to the LDES in LDP specification.

[argahsuknesib]

I have been using the real-time replaying feature of the replayer, where after loading the dataset from the file and sorting the array, I trigger the startAutoPlay command;

curl http://localhost:3001/startAutoPlay

This results in the observations being added into the LDES Stream. However, the replay isn't spec compliant.

At the time of initialization of the LDES Stream, the first container (which is also the inbox) is created.

When replaying historical data, the events are not added into the inbox with a POST request (as they should according to the spec https://woutslabbinck.github.io/LDESinLDP/#adding-member). Instead, new containers are created, containing the events from the file.

A simple solution for this will be to avoid the year-month-date in the historical data file, and just consider the difference between two events and set the starting time of the stream (t0) to be the initialization of the LDES, instead of the historical data from the file.

[svrstich]

I think we need to have a closer look at the NaiveAlgorithm from Wout then. As the only thing that the re-player does, is based on the timestamp difference between two consecutive measurements, calling the function function advanceOneObservation(). This function collects all related (context-)information about the observation that needs to be played, and sends this in an array to Wout's Algorithm:

await naiveAlgorithm(lilURL, finalResources, treePath, bucketSize, config, prefixes, session, loglevel);

Or am I mistaken?

[argahsuknesib]

We can also change the timestamp of the events read from the file (not sure how it accounts for the real-ness of the data, but should be fine), and just take in account the difference. So the first event timestamp being read, for example if it is 2023-11-05T03:04:34.134Z we change it to Date.now() and then the difference between the file events, we keep adding it to the Date.now() variable?

Or, maybe ditch the naiveAlgorithm and just POST to the inbox and update metadata? But then again yeah, when a new inbox should be created would have to be decided. I think we should look into a different naiveAlgorithm's implementation for a real-time dataset as well.

[svrstich]

But are these not two separate things? On the one hand we have the fact that the observations are not stored in the Inbox, but on the other hand we have the issue about the timestamp. So my question is: what's the goal of the re-player??

1. replaying an existing dataset with the original timestamps
2. artificially creating a new dataset with up-to-date timestamps, but based on a historical dataset

[argahsuknesib]

In my opinion, the goal of the replayer is to replay the events to the pod as they are happening right now. Therefore, original historical timestamps don't make sense. The important aspect for the streaming data being replayed should be the value, and the frequency of the stream dataset being replayed. If we replay the dataset with an artificial timestamp, the most recent events will be published to the inbox in a spec-compliant way.

[svrstich]

OK, fair enough, but that was not how I interpreted it at first. Let's check with Femke before committing effort into this RFE.

[argahsuknesib]

Yes sure. I will tag her here in this discussion. CC : @fongenae

[svrstich]

OK, I think we can fairly quickly resolve this issue .. as in .. we can make it an option whether or not you want a real replay, or whether you want to use the loaded dataset as a template, with the timestamps adapted to reflect current day and time. I'll get it done! Give me some time, though ;-)

[argahsuknesib]

Yes definitely, thanks for the help stijn!