search

mw99 / DataCompression

Swift libcompression wrapper as an extension for the Data type (GZIP, ZLIB, LZFSE, LZMA, LZ4, deflate, RFC-1950, RFC-1951, RFC-1952)
Apache License 2.0
287 stars 57 forks source link

Test Failure #17

Closed mkeiser closed 5 years ago

mkeiser commented 5 years ago

Hey, first of all thanks for this great library! I use this in an app of mine- it seems to work very well.

However, I suddenly seem to get test failures (macOS 10.4.4, Xcode 10.2).

The falling test is testRandomDataBlob, specifically the .lzfse decompression test. Apples data compression lib seems to make it hard to find out what the actual error is though. Do you have any idea whats up with that?

Btw the way this test is set up makes it look like more tests are failing after a first test failure. However really only lzfse decompression fails.

mw99 commented 5 years ago

Thanks!

Hmnnn.... OK. That is not very nice. I admit that the tests are bad structured. Testcases separated for each compression method would be better to catch that. My idea was to just test my wrapper code extensively and just assume that Apples code works. Well we see how good that worked out...

Generating a big blob of random data is also not very reproducible. I also had other complains that Apple's lz4 implementation is not compatible to other implementations. And my test showed that 50% of the time it doesn't compress the data at all. But obviously I can't do anything about Apples code.

I will restructure the tests and will start to use random data with a static seed to be at least reproducible in the next release. But I'm afraid that is all I can do for you. My general advice is: Just stick with inflate/deflate (zlib). I has proven itself in the last decades and just works while offering a good ratio of time to compression rate.

(I will close this issue when I have restructured the unit tests)

mkeiser commented 5 years ago

My idea was to just test my wrapper code extensively and just assume that Apples code works.

Ah, so you think the problem really is with Apples library? That would be quite worrying. 😧

My general advice is: Just stick with inflate/deflate (zlib).

I’m currently using gzip, hoping that’s solid too.

And my test showed that 50% of the time it doesn't compress the data at all.

I’m not sure if you’re saying that the library failed to do anything at all, or just that the resulting data is not smaller than the input. If the latter: I’m not a specialist in this area, but I think it is a basic property of information theory that random data generally can not be compressed. (Edit: it occurs to me you were probably not talking about the random data test, if so, disregard 🙂)

Thanks for your quick reply and the great work you did with his library!

mw99 commented 5 years ago

Once again thank you very much for your help. I also added better test cases in the 3.5.0 release. Sticking with random data for now though.