WiresmithTech / tedium

3 stars 1 forks source link

Streaming Write API #6

Open JamesMc86 opened 1 year ago

JamesMc86 commented 1 year ago

Something like the advanced/async API in the NI library this would allow you to:

  1. Specify a single type of write for the next set of writes.
  2. Write these at very high performance.

This means we would be able to skip all header checks and assume all data coming in can just be written to disk - essentially maxing out disk performance.

We can use Rusts type system to create a typed writer for this to make the API easy.

JamesMc86 commented 1 year ago

So the reason for this is write now we do produce a new segment on every write.

We probably want something like a "SegmentWriterTime" which wraps the TdmsWriter type

When you first call this it will write the header and note where the next_segment_offset needs to be written.

It will be able to be called with as many write_data as needed, tracking the length written. When dropped/ended then it will commit the segment length.

JamesMc86 commented 1 year ago

Thinking this through and this could just be the default used everywhere internally.

Imagine this:

In fact we should write FFFFFFF as next segment as this is how the crash detection works. But it might be nice to make this optional so we don't have to do an additional seek unless necessary. Perhaps we do need two modes.