Streaming (C like) API for heatshrink

[jcdubois]

The API for the RUST version of heatshrink differs quite a bit from the philosophy of the C version.

With the C API you can implement a streaming algorithm where data is processed chunk by chunk (think network packet) and you can therefore compress or uncompress data without having to allocate memory for the whole image. For example on one side you can receive data as a stream of network packets, compress or uncompress each packet and then write the resulting data to storage media as list of individual blocks.

This is not possible with the current RUST API which expect the source and the destination to be 2 memory blocks big enough to contain the all data image. It is therefore not possible to process the data on the go as it needs to be processed at once.

Do you intend to provide some other API more compatible with streaming scenarios which seems to make a lot of sense for embedded targets with limited amount of memory?

Obviously the current API is quite simple (compared to a streaming API) but it seems that it requires memory resources that might be missing to embedded targets.

[snakehand]

There is no real contradiction between memory size limitation and a streaming API, the only thing that would be required is that the window needs to be separated from the input / output buffers, which requires more memory. Another requested feature is faster compression time, and this would make sense to implement with a separate data structure associated with the input window, that also includes lookup tables. So it would make sense to combine the streaming API and faster compression speed in the same development cycle.

[jcdubois]

Well I have a heatshrink RUST implementation that follow more closely the C implementation logic including the lookup table (so I guess the performance should be on par. I should verify this).

The caveat is that this implementation is (for now) limited to the heatshrink parameters recommended for embedded target (same limitation as if you were compiling the C implementation in static mode): window_size = 8, lookahead_size = 4.

From there, it is easy to implement your RUST API on top of it (with the limitation that you cannot change the heatshrink parameters from the embedded ones).

[jcdubois]

I provided the actual state of my tentative port of the C version at https://github.com/jcdubois/heatshrink-rs.

This port is certainly still too much C like.

Performance is still a bit behind the C version.

However this could be used as a reference point for discussion.