make DA blocks transient

[crotwell]

Thinking more about this, I do not like the multiple repeated data blocks in records. I think that something like the DA block is a really good idea for low latency transmission, but not for archiving and data center output.

For real time, the protocol for the sender would be:

1. Send Header
2. Repeat send DA blocks
3. Send TE block
4. Repeat from 1.

The receiver would:

1. Receive Header, hold in memory
2. Repeat on receive transient DA block:
   • += number of samples
   • += number of bytes
   • append data bytes
3. Recieve TE block, append, close record
4. Repeat from 1.

The archive storage would then look more like the 0622 spec with a header, one data section and a footer.

Consider a low latency case were we are sending a DA block every 0.1 seconds with 10 samples from a 100sps channel, not unreasonable if early warning is really going to work. The final record would have 6 bytes of DA block "header" bloat for the data which might fit into 10 bytes!

Over the wire, this makes sense as you want the low latency, but in the archive or for an end user it is terrible. Of course you can concatenate DA blcoks, but as soon as you do that why would you not concatenate all DA blocks within a record?

[crotwell]

Further issue is that some things in the fixed header may not be knowable at the time it is sent. For example:

• Clock locked or unlocked during record may change "Flags"
• Calibration starts during a record, also changes Flags
• Actual sample rate due to clock slewing, back-calculated from (end-start)/(numSamples-1), but end can't be known until end of record

Perhaps a better protocol for low latency would be:

1. Send header with indicator 'MI' for miniseed incomplete
2. Repeat sending DA blocks, receiver appends to existing record
3. Send Extra Header block, 'EH'
4. Resend fixed header with indicator 'MS' with any updates (, receiver replaces original with updated header).
5. Send next MI header to start next record.

This allows the receiver to do early warning processing on the data in a low latency manner, but makes the final archived record as good and complete as it would be if the logger waited to send the entire record as a single transmission.

My idea may not be the correct way forward, but I do feel it is important if we intend the structures to be used in a low latency protocol to think in detail about exactly how they would be used, and probably to also include the proposed protocol in the document at least in a big picture manner.

[andres-h]

Clock locked or unlocked during record may change "Flags" Calibration starts during a record, also changes Flags

Then start a new record. But you wouldn't have the problem with chunks format in the first place.