JuliaIO / BufferedStreams.jl

Fast composable IO streams
MIT License
42 stars 20 forks source link

Source should be permitted to push data onto upstream BufferedInputStream #45

Open sambitdash opened 7 years ago

sambitdash commented 7 years ago

The source IO interface today is expected to implement a readbytes! interface. The readbytes! interface is a pull interface. Which essentially means the BufferedInputStream can demand the source the amount of bytes it needs to provide.

Imagine a source like a codec. The source is not aware of the data size till the data is decoded. But the BufferedInputStream request is for a number of bytes that is lesser than the data decoded. In this case, the source needs to maintain its own housekeeping buffer to retain number of bytes. While, this is there in most codec implementations, ideally as a buffer mgmt system BufferedInputStream should be flexible to either request for a size from the source or let the source to request for additional bytes. That way Sources will not have to maintain their own housekeeping with buffers.

One example can be ReadFile call in Windows API, where passing NULL for bytes read will provide the size of the buffer required.