Fail to decompress buffer

[ajaybhargav]

This is really strange, I am able to compress but fail to decompress a "specific" set of data buffer. I am using static linking with following configuration:

#define HEATSHRINK_STATIC_INPUT_BUFFER_SIZE 512
#define HEATSHRINK_STATIC_WINDOW_BITS 10
#define HEATSHRINK_STATIC_LOOKAHEAD_BITS 3

uint8_t compressed[] = {
0x80, 0x80, 0x35, 0x45, 0x01, 0x7B, 0xA9, 0x07, 0x01, 0xF4, 0x01, 0xB0, 0x00, 0xE0, 0xA0, 0x0D, 
0x45, 0x80, 0x5E, 0x82, 0x00, 0x35, 0xA8, 0xB7, 0x4F, 0x7E, 0x10, 0x89, 0x10, 0x03, 0x33, 0x3A, 
0x2C, 0xBA, 0x21, 0x7C, 0x04, 0x87, 0xE1, 0x42, 0xB2, 0x7C, 0xE7, 0xFF, 0x88, 0x70, 0x26, 0x20, 
0x00, 0xC2, 0xE0, 0x70, 0x10, 0x16, 0x82, 0x9F, 0x00, 0x1C, 0x00, 0x70
};

/* Actual input */
uint8_t decompressed[] = {
0x01, 0x00, 0x00, 0x00, 0x45, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xD4, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0xA8, 0x6E, 0x3D, 0xF0, 0x08, 0x22, 0x3D, 0xF0, 0x33, 0x45, 0x2E, 0x10, 0x7C, 0x01, 0x00, 0xF0, 0x42, 0x64, 0xF3, 0x3F, 0xF8, 0x0E, 0x00, 0x05, 0x05, 0x05, 0x0B, 0x03, 0x01, 0x64, 0xF3, 0x3F, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

I have also attached my test application. So problem is:

If I try to "only" decompress the compressed buffer, I run into a never ending loop of poll. However if I first compress the "decompressed" buffer and then decompress in the same application everything works. I am not able to understand why?

to check both usecases, make if 0 @line 202 to 1

test_heatshrink.txt

PS: Rename file to .c

[silentbicycle]

Thanks for providing a detailed test set. I will look into this over the weekend. One more detail: what version of heatshrink are you seeing this behavior in?

[silentbicycle]

I can reproduce the issue with the code you sent. (Again, thanks.) It looks like your test code is getting stuck in the do-while loop on lines 85-91 forever, because you're calling heatshrink_decoder_poll with an out_buf_size argument of 0. It returns HSDR_POLL_MORE, indicating that it needs a fresh buffer. It can't do any further decompression because you've just handed it a buffer with 0 bytes of space left. This isn't an error condition (dpres < 0), but it also doesn't cause any progress, and you keep retrying forever.

Typically, after filling the buffer, its contents would get flushed out to a file / socket / flash ROM / etc., and then heatshrink_decoder_poll would be called with the full buffer available again.

(It looks like when you're also compressing, the buffering is different, so it's not running into the issue.)

This isn't a bug, it's a misuse of the API, but it's a something that could be documented more clearly. Any suggestions?

[ajaybhargav]

Thanks for the pointer... I found some issue in the test code. Issue: outsize was lesser than the required size. inlen is 60 so according to following statement: outsize = inlen + inlen / 2 + 4; outsize was always lesser than decompressed size (100 bytes). Hence it loops forever

I did a test by setting outsize to 100 bytes since my final data is 100 bytes as I know. so changing above to: outsize = 100; also loops forever, it only works if I set it to 101. So the question why 1 byte extra?

PS: I am using latest git version.

[ajaybhargav]

Did you get a chance to test with outsize set to 100?

[silentbicycle]

Not yet.

[ajaybhargav]

ok.. just want to summarize, if buffer length == decompressed data length algorithm goes into infinite loop (waiting for buffer I guess).

[silentbicycle]

I plan on confirming this (and if so it'll get a bugfix release), but I'm been very busy/traveling so it may be a couple days.

[silentbicycle]

I think it's still due to misuse of the API. You aren't calling heatshrink_decoder_finish until execution gets outside the loop the code is stuck in, but it can't make progress without extra buffer space (even though it shouldn't be necessary) because it doesn't know that it has all the input. (Except at the end of the stream, it always needs to work with a consistent amount of buffer space - otherwise, working in different chunk sizes would lead to incompatible results.)

The test and CLI code your appears to be based on gets around this problem by putting the output buffer's contents into a file/output stream and resetting it, and by notifying when the end of input is reached.

Essentially, if you call heatshrink_decoder_poll and get a response of HSDR_POLL_MORE, you must flush your output buffer before calling it again.

[ajaybhargav]

Ok, I will take a look at the APIs again. There is little documentation so I might have missed something. My application is based on sample codes provided and I used them with little modification.

[ajaybhargav]

Replying too late :) After you explained about API, I made changes to code and found that I was not using it correctly. However I blame ( :p ) the confusing test code for this as I followed it and not much documentation available for this library.

Thanks for your input I am able to get it working perfectly. My example code can be a good starting point for those looking for a simple compress/decompress function. Can I submit it in this library for anyone to use it?