search

oduwsdl / MemGator

A Memento Aggregator CLI and Server in Go
https://memgator.cs.odu.edu/api.html
MIT License
55 stars 11 forks source link

Streaming aggregated TimeMap #95

Open ibnesayeed opened 7 years ago

ibnesayeed commented 7 years ago

It may be useful to have the aggregator stream TimeMaps as they accumulate from various upstream archives. Sorting in this case will not be needed (or even practical). This can be utilized in some visualizations or tools like Mink where the number may dynamically change as more and more responses arrive from upstream archives. From the implementation perspective, chunked encoding could be one of many other ways to transmit the data to the client.

machawk1 commented 7 years ago

@ibnesayeed This would be very useful for tools like Mink where a partial result is often better than waiting for a comprehensive result if the latter is not needed by the user.

ibnesayeed commented 7 years ago

The ORS/CDXJ format will be a good candidate for the stream serialization. Perhaps we should only support that format for streaming.

machawk1 commented 7 years ago

@ibnesayeed Agreed, and this highlights one of the benefits of using CDXJ over Link (or JSON) since a partial TimeMap is still valid and useful.

ibnesayeed commented 7 years ago

There is a potential of utilizing the chunk extension to provide metadata about each batch of response that might contain the name of the archive and number of mementos after the batch size.

2a7; provider=IA; memento-count=15;
... ORS/CDXJ lines...