Improving handling of a trickle of data into tyger buffer write

[yuliadub]

Another attempt for trickle of data, this time with go routine:

• if time window is supplied, a go routine is kicked off that will try to read from io
• the reader will return what is currently there to read, which should then be written by tyger for the flush interval duration
• this does unblock being able to write without having to wait for sleep to complete
• reading wont happen again until the buffer is flushed

  ---

This is a first draft for trickle of data write for tyger - instead of setting a block size for writing buffers, the user would instead provide a time-window on how often the writes should happen (time-window would define that writes should happen every X seconds) (https://github.com/microsoft/tyger/issues/93)

Main thing I ran into is that read (both in io or bufio) wants you to specify the exact buffer size to read into vs being able to just read what is current present in io.Reader. If using ReadAll, the reader will wait for an EOF before starting to read. I couldn’t find a way to see the current size of io.Reader without actually reading the data to begin, which also wait for an EOF before returning the size.

The approach I choose was to keep reading single bytes of data from io.Reader and append them to an existing buffer[] until the defined time window is over. Then take what was read and write that to a buffer same as with block size. Repeat until EOF.

I added the simple script I used for testing just to have an example - will remove it once this PR is no longer a draft.

What this pr doesn’t do yet but should:

• You can provide both block size and time window, which likely doesn’t make sense. Need to add functionality that only one of these can be provided
• No specific tests
• Likely better error handling if something goes wrong (bytes too short exception for example)
• Probably other things that are escaping my mind in this exact moment

[johnstairs]

If you do something like

tyger buffer gen 1G | tyger buffer write $(tyger buffer create) --flush-interval 10s

You end up with blobs that are 64KiB, whereas they should be 4MiB.

[yuliadub]

going to close for now - apologies I have not had time to complete this. I am hoping to get more time to work on it once I am back from vacation in 2 weeks if it would still be relevant.