Add brunsli JPEG compressor as a transform

[M-Gonzalo]

Hi Frederic! Do you think it might be possible for you to add a new transform for jpg files using brunsli? Especially if it can detect them inside compounded data, like a tar archive.

I'm available for testing! BTW, reporting that kanzi is working flawlessly on ARM aarch64, compiled with clang 👍👌

[flanglet]

Hi Gonzalo, First, thanks for testing kanzi on ARM64. I rarely compile and test on this platform, so it is great to see everything is working. Realiability and portability are 2 important goals. BTW, if you are testing, always use the releases because the tip is sometimes broken. Regarding brunsli, it would be possible to integrate as a transform of course. Now, I am not sure what it would bring since it would not be different from a standalone brunsli. Also, brunsli already contains model and entropy coding, so it would really fit as a transform (maybe entropy coding could be differed to kanzi). Finally, to address the tar comment, it is hard problem to decide how to handle archives. If a tar is made of thousands of small files, then. compressing one-by-one can be very costly in terms of speed vs. compiling all at once (or block by block). So, for now, I have decided to use the latter option (the type detection still applies on a per-block basis).

[M-Gonzalo]

I am not sure what it would bring since it would not be different from a standalone brunsli.

You're right, not by itself. But I'm thinking kanzi's got the potential to become somewhat of a universal pre-processor. Like a better, more modern precomp. So adding to the mix something like WavPack for audio, or packMP3, could be very helpful as a previous stage to lzma2.

it is hard problem to decide how to handle archives.

I agree! Ideally, a tar could be parsed and decomposed, but I'm thinking more like just being on the lookout for headers or magic numbers particular to JPG images, for example. The use case I have in mind is a fast in-memory pipe between tar | kanzi | someCompresor. Again like precomp does but better. Am I making sense here?

[flanglet]

I see where you are going with this. Discarding the entropy part to just use the tranforms with another entropy codec is not my plan. I would rather improve the entropy stage if needed. Also, I would consider adding these kinds of prefilters when I convince myself to build an archiver on top of this compressor (maybe next year). It would fit better in that context.

[M-Gonzalo]

Yes, I understand what you mean and I agree. Perhaps I didn't explain myself clearly. I'm not advocating for striping the entropy coding from anything, just compressing the file/stream with an external library (including its entropy coder if it uses one), and pass that along. Doesn't really matter if there's some more entropy coding down the line in the tool chain, as it will happen anyway if the final codec tries to compress a JPG image, or most multimedia for that matter, which is already compressed.

[flanglet]

I understand your POV but I will close this. I do not want to add this kind of dependency in the compressor (it would be hard to maintain consistency with the Java & Go implementations). I also do not want kanzi to turn into a pre-processor for other compressors. Once I start work on an archiver (only C++), then I am open to adding these pre-processing tools.