search

bp74 / Zstandard.Net

A Zstandard wrapper for .Net
Other
135 stars 26 forks source link

Underlying stream read timeout #10

Closed Brain2000 closed 5 years ago

Brain2000 commented 5 years ago

When calling the Read( ) function, as it reaches the last frame in a stream, if a small number of bytes are requested it will hang for the underlying stream's timeout duration if the underlying stream is of a blocking type such as a network stream. This timeout pause happens once, which then sets the dataDepleted Boolean to true, and then the remainder of the bytes will flush out from the internal buffers used in ZSTD_decompressStream. Unfortunately, this timeout pause will happen even if the caller knows the exact number of bytes that will be read.

This pause can be mitigated by pulling all the bytes from the internal buffers first to complete a frame, and only continue to Read the underlying stream to start a new frame when the uncompress write position remains at 0.

An example of this would be streaming to the MSSQL VDI interface from an underlying network stream. SQL knows the exact number of bytes that it needs to pull, yet the timeout occurs on the last frame because the VDI is requesting 512 bytes at a time near the end.

I made a mod to ZstandardStream.cs where the underlying stream will Read only after the current frame has been completed and flushed, thus avoiding this timeout. If you're interested, I can submit my change with a pull request for your review.

bp74 commented 5 years ago

Hi, thanks for the report. It would be great to get a pull request so i can take a look at the fix.

Brain2000 commented 5 years ago

I created it. This is my first pull request on Github. I hope I performed it correctly.

Brain2000 commented 5 years ago

@bp74 This week I ran around 40 TB of streaming data with the change in the pull request. I have not received a single timeout, and I haven't had any errors as a result of the change.