Gappy stream issue

[jdassink]

Hi

I am experimenting with a ringserver setup in combination with a sensor that does not provide miniseed data natively. The sensor provides files per minute. Each minute, a new file is opened that stores the sample for that minute. It is possible to access the file for the current minute.

The ringserver setup works well when I provide ringserver "complete" minute files in the /data/mseedscan directory that is listed in the ringserver.conf file

-----------------------
RingDirectory /data/ring
DataLinkPort 16000
SeedLinkPort 18000
ServerID "My first Seedlink server"
TransferLogDirectory data/tlog
TransferLogRX 0
MSeedScan /data/mseedscan StateFile=/data/scan.state InitCurrentState=n
-----------------------

Now, I am trying to reduce the latency as much as possible, and for that I am also reading out the "current minute" file from the sensor, say two times per minute. For that purpose I have written some scripts that successfully write miniseed files into the /data/mseedscan folder. When I run "msi -T ", I can indeed see that the last file is growing, until it has the complete number of samples. So, there are no issues on that side.

However, when I let ringserver scan these files, it seems to become somewhat confused! Running slarchive on the seedlink server provides very gappy data for some reason.

How to solve this?

[chad-earthscope]

Hi @jdassink. Hmmm, according to your description that should work just fine.

Are there any errors or other indications in the ringserver log file? Maybe if you turn up the verbosity of ringserver you can see something that is a clue?

As a suggestion, if you're goal is to reduce latency I recommend considering writing a program that creates miniSEED and submits the records to ringserver via the DataLink protocol directly, to avoid using the file system. This is a bit more complicated than writing to files (needing a persistent process), but avoids the scanning of files and gives you total control of when data are added and streaming from ringserver.

[jdassink]

Thanks @chad-earthscope, I like your suggestion. How can I submit MiniSEED records via the Datalink protocol?

[crotwell]

You can send/receive datalink in python with simpledali. Example is sendMSeedPacket.py

[chad-earthscope]

@crotwell 's suggestion is great for the Python.

Some other options would be to write a submission program in C. Here the the two most prominent examples: slink2ew (SeedLink client for Earthworm) slink2orb (SeedLink client for Antelope)

Both of those program also create miniSEED out of packets of data in those systems, and use a small rolling buffer in memory in order generate filled miniSEED records. Obviously this adds a bit of latency. There is always going to be a tradeoff between low latency (smaller packets) and what you want to archive long term (bigger packets), but what is to be avoided if at all possible is padded data, which is bad from most perspectives.

To have a program submitting data via DataLink to a ringserver request the complexity of managing a separate process to do that work, presumably it's something you can add to whatever is managing the ringserver process.