search

JuliaIO / TranscodingStreams.jl

Simple, consistent interfaces for any codec.
https://juliaio.github.io/TranscodingStreams.jl/
Other
85 stars 28 forks source link

Add optional `pledgeinsize` function to transcoding protocol #239

Closed nhz2 closed 6 days ago

nhz2 commented 1 week ago

The pledgeinsize(codec::C, insize::Int64, error::Error)::Symbol method is used when transcode is called to tell the codec the total input size. This is called after startproc and before process. Some compressors can add this total input size to a header, making expectedsize accurate during later decompression. By default this just returns :ok. If there is an error, the return code must be :error and the error argument must be set to an exception object. Setting an inaccurate insize may cause the codec to error later on while processing data. A negative insize means unknown content size.

Ref: https://github.com/JuliaIO/CodecZstd.jl/pull/58

I think this is a nicer alternative to #215

mkitti commented 1 week ago

I wonder if it would be appropriate to call this sizehint!.

nhz2 commented 1 week ago

I chose the name to match the style of the other functions in the transcoding protocol. This is also slightly stronger than a hint since with Zstd setting the wrong size will cause an error.

mkitti commented 1 week ago

I guess the convention here is not to use !?

Could we make it clearer the order in which these functions will be called?

Also do we have a mechanism to pledge the size even in streaming mode?

nhz2 commented 1 week ago

I added some more details to the order of the function calls. The transcoding protocol can be used outside of the transcode function since it is part of the stable API, though care has to be taken because these functions are potentially memory-unsafe.

nhz2 commented 6 days ago

@mkitti I added some examples of how pledgeinsize could work in one of the test codecs.