Read range of packets

[telecoder]

Hi,

Is there a way to request all packets within a time range?, this would be something similar to the TIME command from SeedLink.

Thanks,

[chad-earthscope]

You could using dl_position_after(), i.e. the POSITION AFTER hptime protocol command, to start a reader based on a time and then stop reading when the desired end time is reached.

[telecoder]

ok, lets say I want to download a portion of data for stream A, so I use MATCH for that stream and then POSITION AFTER to get an initial pktid, how would I request the next packet for that stream?, I don't see anything preventing consecutive packet ids to belong to different streams.

[chad-earthscope]

A MATCH is a subscription, or filter, on the connection. Once you set MATCH and/or REJECT they will be used to filter packets for the connection until they are changed.

After submitting MATCH and POSITION one would commonly issue STREAM to start streaming data flow.

If you have not found it yet, here is the protocol documentation: https://iris-edu.github.io/libdali/datalink-protocol.html

[chad-earthscope]

It should also be noted that fetching time windows of data is not a primary use case for DataLink or ringserver or even SeedLink (IMHO). These are streaming protocols and servers meant to continuously transmit data as it is submitted to the server to clients. There are subtle mismatches between streaming capability and implied functionality of time-based requests. For example, when requesting a time window from a streaming server when does it think the request is done? In most SeedLink implementations this is implemented to be when a packet is received that is after the requested window. But if the data are not in time order this means the "time window" request may be incomplete.

I recommend against using DataLink and similar streaming protocols for fetching time windows of data. Instead, use it to stream data; the stream can be interrupted and resumed as you wish.

[telecoder]

Thanks for your comments, really appreciated.

In most SeedLink implementations this is implemented to be when a packet is received that is after the requested window. But if the data are not in time order this means the "time window" request may be incomplete.

I was thinking about implementing a FDSN Dataselect facade for ringserver, but then I guess this invalidates it, since it's common for real time data to have packets out of order.

I think there is a valid use case for queries for time windows of data that is kept in ram; in real time monitoring applications it could be common to frequently reprocess recent hours of data. I would rather be hitting ringserver than an Earthworm or a Seiscomp instance.

[chad-earthscope]

I was thinking about implementing a FDSN Dataselect facade for ringserver, but then I guess this invalidates it, since it's common for real time data to have packets out of order.

Ha! I've thought quite a bit about putting an fdsnws-dataselect API on ringserver. The architecture is not really designed for that but it's not impossible and could fill a niche functionality.

For what it's worth, I do not think it's common for streaming data to be out of order. The issue is that in a streaming scenario the implication for FIFO-like servers is not a big deal, but the implication for fdsnws-dataselect is that nuances of streaming data should not matter, implying extra work to do the most complete job.

I think there is a valid use case for queries for time windows of data that is kept in ram; in real time monitoring applications it could be common to frequently reprocess recent hours of data. I would rather be hitting ringserver than an Earthworm or a Seiscomp instance.

That is a good use case and makes sense.

[telecoder]

it took me a little while but here is an early version.

gerbera