search

hapi-server / data-specification

HAPI Data Access Specification
https://hapi-server.org
23 stars 7 forks source link

Open end streaming of data #167

Open jwittke opened 1 year ago

jwittke commented 1 year ago

Is it possible to specify a server request that initiate a endless data stream to the user? Will this be part of any future release?

rweigel commented 1 year ago

If you mean, "can one request data over the full time range?", then it is possible with some servers. We do have a maxRequestDuration that servers can use to tell the user that the time range requested is too large to handle.

The Python client allows the user to break up a request into smaller chunks so as to not hit the maxRequestDuration limitation. In this case, the user will not need to handle the chunking and can pass the full time range.

I don't think we'll ever require servers to handle arbitrarily long time ranges, however.

jwittke commented 1 year ago

Thank you for the comment.

I mean, if I want to include real-time sensor data, which will be continuously produced. For example, I want to download from a time span which covers already produced data from the sensor up to a time in the future. The server will deliver then the data up to the time of the request and doesn't stop delivering while new data gets available, until the end date from the request comes true.

rweigel commented 1 year ago

I think this is possible. One sets the stopDate into the future. If a new record arrives every hour, I would think the socket would time out. The server could modify the Connection Timeout header and then the clients should respect this.

If you have more details, we could set up a demo server that implements the streaming of generated data.

On Mon, Mar 6, 2023 at 7:40 AM Jan Wittke @.***> wrote:

Thank you for the comment.

I mean, if I want to include real-time sensor data, which will be continuously produced. For example, I want to download from a time span which covers already produced data from the sensor up to a time in the future. The server will deliver then the data up to the time of the request and doesn't stop delivering while new data gets available, until the end date from the request comes true.

— Reply to this email directly, view it on GitHub https://github.com/hapi-server/data-specification/issues/167#issuecomment-1456061566, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUQ57QJTXOE75HCJJUUYCDW2XLKRANCNFSM6AAAAAAVQ63IQ4 . You are receiving this because you commented.Message ID: @.***>

jvandegriff commented 1 year ago

I think the intent for this is to support a more continuous kind of connection, more of a polling capability, or a kind of streaming service that you subscribe to and new data shows up when available.

HAPI does not have this now - it's just HTTP requests, which have time limits, so they can't support long-term stream connections.

There might be a way to add a streaming service on top of a HAPI server that polls the underlying HAPI server periodically, and then offers a more streaming kind of connection to clients.

rweigel commented 1 year ago

Sounds like WebSockets.