API for Buffer - Githubissues

JuliaIO / TranscodingStreams.jl

Simple, consistent interfaces for any codec.

https://juliaio.github.io/TranscodingStreams.jl/

Other

85 stars 28 forks source link

API for Buffer #202

Open jakobnissen opened 5 months ago

jakobnissen commented 5 months ago

I was taking with @nhz2 about how Automa (and presumably other packages) use internals from TranscodingStreams, and how it would be nice to make it stop doing that. However, TS was created by the same author as Automa, and the two packages' internals are entangled. So, for Automa to not rely on TS internals, TS needs to offer these things as API. In particular:

Access to the .data field of a TranscodingStream, either as a Vector{UInt8} or potentially Memory{UInt8} going forward
Get and set the .markpos and .bufferpos fields of the read Buffer of a stream
Get the .marginpos of the read Buffer of a stream

Ideally, the Buffer object can also be obtained so the user can manipulate the Buffer directly.

Another approach is for Automa to stop using TranscodingStreams, and instead use something like BufferedStreams.jl. However, I believe TranscodingStreams might have originally been implemented to support Automa.

nhz2 commented 5 months ago

Thank you for making this list as I don't understand the internals of Automa. I think it is fine to keep the Buffer type and .data, .markpos, .marginpos, and .bufferpos fields as they are very well documented in src/buffer.jl

Also, do you know if Automa is using the unread or unsafe_unread functions? I would like to better specify how those functions interact with position and mark.

Lastly, does Automa use any of the write functions?

jakobnissen commented 5 months ago

Automa does not use unread or unsafe_unread. It does use Base.write, but not writing to a TranscodingStream, so it shouldn't be of concern for this package.

From what I can tell, what Automa uses is:

That you can get a Buffer containing the input data by doing stream.state.buffer1 - this could be changed to a getbuffer function if necessary
The fields .markpos, .marginpos and .bufferpos, and their precise meaning
That you can get a Vector{UInt8} by reading the .data field of a Buffer - this could perhaps be changed to a function also, since maybe in the future, this should be a Memory
That a TranscodingStream behaves like an ordinary IO object, e.g. that eof works and returns a Bool. This is perhaps too trivial to mention.