Any way to build unpack-only C-library?

[Ykidia]

Hello! Is there any simple way (i.e. without removing bunch of code) to build a minimal static C-library which is only unpack-capable, i.e. without pack features? Thank you!

[flanglet]

Hi,

There is currently no such option. Is the size of the library an issue ? If you are using the Makefile to build the library, I can add a new target for a decompression library only.

[Ykidia]

In details, I want to try to use kanzi decompressor for linux kernel instead of, say, xz. Because, in some circumstances, kanzi packs Linux kernel (embedded, ARM and MIPS) by 5-10% better than xz. And if kanzi unpacker will not be so bigger than xz unpacker, then I can use it for my kernels builds by default)).

If you are using the Makefile to build the library, I can add a new target for a decompression library only.

Yes, please! But only if you don't mind. I'll be very thankful!

[flanglet]

I have added new targets in the Makefile to build comp/decomp only librairies. You can give it a try (make libdecomp). A few notes: I have not yet split the libapi.cpp file which should be done in the final solution but I want your feedback first. Also, I doubt the resulting library can ever be smaller than liblzma because kanzi has so many more transforms than lzma. Finally, the bitstream can change between 2 releases and is not always backward compatible. So, you can either test with the current code, download the 2.1 sources and use the new Makefile or wait for the 2.2 release (circa Sept 2023). I expect the next release to be pretty much complete so the bitstream is unlikely to change after 2.2.

[Ykidia]

It's wonderful!) But I forgot to say that I am cross compiling. Host system is Linux x86_64, but target ARCH is armv5tej and mips. Tried for now only for arm5tej. At first, I added $(CROSS_COMPILE) to CXX in Makefile. And changed ARCH := <...> to ARCH ?= <...> because I am building for other system. Build command after changes looks like this: ARCH=armv5tej CROSS_COMPILE=arm-none-eabi- make libdecomp I got then some errors about mutex and condition_variable from concurrent.hpp. I think if concurrency is about multithreading then I don't need it (at least for now) and disabled CONCURRENCY_ENABLED define in concurrent.hpp. Eventually, I have fewer errors:

ARCH=armv5tej CROSS_COMPILE=arm-none-eabi- make libdecomp
arm-none-eabi-g++ -c -std=c++11 -Wall -Wextra -Os -fPIC -DNDEBUG -pedantic -march=armv5tej -fno-rtti io/CompressedInputStream.cpp -o io/CompressedInputStream.o
In file included from io/CompressedInputStream.hpp:22,
                 from io/CompressedInputStream.cpp:18:
io/../concurrent.hpp: In member function 'int atomic_int::load(int) const':
io/../concurrent.hpp:219:30: warning: unused parameter 'mo' [-Wunused-parameter]
  219 |                 int load(int mo = memory_order_relaxed) const { return _n; }
      |                          ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp: In member function 'void atomic_int::store(int, int)':
io/../concurrent.hpp:220:39: warning: unused parameter 'mo' [-Wunused-parameter]
  220 |                 void store(int n, int mo = memory_order_release) { _n = n; }
      |                                   ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp: In member function 'bool atomic_bool::load(int) const':
io/../concurrent.hpp:249:31: warning: unused parameter 'mo' [-Wunused-parameter]
  249 |                 bool load(int mo = memory_order_relaxed) const { return _b; }
      |                           ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp: In member function 'void atomic_bool::store(bool, int)':
io/../concurrent.hpp:250:40: warning: unused parameter 'mo' [-Wunused-parameter]
  250 |                 void store(bool b, int mo = memory_order_release) { _b = b; }
      |                                    ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp: In member function 'bool atomic_bool::exchange(bool, int)':
io/../concurrent.hpp:251:50: warning: unused parameter 'mo' [-Wunused-parameter]
  251 |                 bool exchange(bool expected, int mo = memory_order_acquire) {
      |                                              ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp: In member function 'int kanzi::CompressedInputStream::_get(int)':
io/CompressedInputStream.cpp:272:30: error: reference to 'memory_order_relaxed' is ambiguous
  272 |             if (_closed.load(memory_order_relaxed) == true)
      |                              ^~~~~~~~~~~~~~~~~~~~
In file included from /home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/atomic:41,
                 from io/../concurrent.hpp:32:
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:80:7: note: candidates are: 'std::memory_order std::memory_order_relaxed'
   80 |       memory_order_relaxed,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:204:19: note:                 'const int memory_order_relaxed'
  204 |         const int memory_order_relaxed = 0;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp: In member function 'int kanzi::CompressedInputStream::processBlock()':
io/CompressedInputStream.cpp:356:38: error: reference to 'memory_order_acquire' is ambiguous
  356 |     if (!_initialized.exchange(true, memory_order_acquire))
      |                                      ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:82:7: note: candidates are: 'std::memory_order std::memory_order_acquire'
   82 |       memory_order_acquire,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:205:19: note:                 'const int memory_order_acquire'
  205 |         const int memory_order_acquire = 2;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp:369:52: error: reference to 'memory_order_relaxed' is ambiguous
  369 |             const int firstBlockId = _blockId.load(memory_order_relaxed);
      |                                                    ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:80:7: note: candidates are: 'std::memory_order std::memory_order_relaxed'
   80 |       memory_order_relaxed,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:204:19: note:                 'const int memory_order_relaxed'
  204 |         const int memory_order_relaxed = 0;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp: In member function 'void kanzi::CompressedInputStream::close()':
io/CompressedInputStream.cpp:513:32: error: reference to 'memory_order_acquire' is ambiguous
  513 |     if (_closed.exchange(true, memory_order_acquire))
      |                                ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:82:7: note: candidates are: 'std::memory_order std::memory_order_acquire'
   82 |       memory_order_acquire,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:205:19: note:                 'const int memory_order_acquire'
  205 |         const int memory_order_acquire = 2;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp: At global scope:
io/CompressedInputStream.cpp:547:5: error: reference to 'atomic_int' is ambiguous
  547 |     atomic_int* processedBlockId, vector<Listener*>& listeners,
      |     ^~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/atomic:1077:41: note: candidates are: 'typedef struct std::atomic<int> std::atomic_int'
 1077 |   typedef atomic<int>                   atomic_int;
      |                                         ^~~~~~~~~~
io/../concurrent.hpp:209:15: note:                 'class atomic_int'
  209 |         class atomic_int {
      |               ^~~~~~~~~~
io/CompressedInputStream.cpp:547:5: error: 'atomic_int' has not been declared
  547 |     atomic_int* processedBlockId, vector<Listener*>& listeners,
      |     ^~~~~~~~~~
io/CompressedInputStream.cpp:544:1: error: no declaration matches 'kanzi::DecodingTask<T>::DecodingTask(kanzi::SliceArray<unsigned char>*, kanzi::SliceArray<unsigned char>*, int, kanzi::uint64, short int, int, kanzi::InputBitStream*, kanzi::XXHash32*, int*, std::vector<kanzi::Listener*>&, const kanzi::Context&)'
  544 | DecodingTask<T>::DecodingTask(SliceArray<byte>* iBuffer, SliceArray<byte>* oBuffer, int blockSize,
      | ^~~~~~~~~~~~~~~
io/CompressedInputStream.hpp:121:8: note: candidate is: 'kanzi::DecodingTask<T>::DecodingTask(kanzi::SliceArray<unsigned char>*, kanzi::SliceArray<unsigned char>*, int, kanzi::uint64, short int, int, kanzi::InputBitStream*, kanzi::XXHash32*, atomic_int*, std::vector<kanzi::Listener*>&, const kanzi::Context&)'
  121 |        DecodingTask(SliceArray<byte>* iBuffer, SliceArray<byte>* oBuffer, int blockSize,
      |        ^~~~~~~~~~~~
io/CompressedInputStream.hpp:106:10: note: 'class kanzi::DecodingTask<T>' defined here
  106 |    class DecodingTask FINAL : public Task<T> {
      |          ^~~~~~~~~~~~
io/CompressedInputStream.cpp: In member function 'T kanzi::DecodingTask<T>::run()':
io/CompressedInputStream.cpp:577:52: error: reference to 'memory_order_relaxed' is ambiguous
  577 |         const int taskId = _processedBlockId->load(memory_order_relaxed);
      |                                                    ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:80:7: note: candidates are: 'std::memory_order std::memory_order_relaxed'
   80 |       memory_order_relaxed,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:204:19: note:                 'const int memory_order_relaxed'
  204 |         const int memory_order_relaxed = 0;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp:601:78: error: reference to 'memory_order_release' is ambiguous
  601 |             _processedBlockId->store(CompressedInputStream::CANCEL_TASKS_ID, memory_order_release);
      |                                                                              ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:83:7: note: candidates are: 'std::memory_order std::memory_order_release'
   83 |       memory_order_release,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:206:19: note:                 'const int memory_order_release'
  206 |         const int memory_order_release = 3;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp:606:78: error: reference to 'memory_order_release' is ambiguous
  606 |             _processedBlockId->store(CompressedInputStream::CANCEL_TASKS_ID, memory_order_release);
      |                                                                              ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:83:7: note: candidates are: 'std::memory_order std::memory_order_release'
   83 |       memory_order_release,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:206:19: note:                 'const int memory_order_release'
  206 |         const int memory_order_release = 3;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp:664:78: error: reference to 'memory_order_release' is ambiguous
  664 |             _processedBlockId->store(CompressedInputStream::CANCEL_TASKS_ID, memory_order_release);
      |                                                                              ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:83:7: note: candidates are: 'std::memory_order std::memory_order_release'
   83 |       memory_order_release,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:206:19: note:                 'const int memory_order_release'
  206 |         const int memory_order_release = 3;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp:670:78: error: reference to 'memory_order_release' is ambiguous
  670 |             _processedBlockId->store(CompressedInputStream::CANCEL_TASKS_ID, memory_order_release);
      |                                                                              ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:83:7: note: candidates are: 'std::memory_order std::memory_order_release'
   83 |       memory_order_release,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:206:19: note:                 'const int memory_order_release'
  206 |         const int memory_order_release = 3;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp:705:78: error: reference to 'memory_order_release' is ambiguous
  705 |             _processedBlockId->store(CompressedInputStream::CANCEL_TASKS_ID, memory_order_release);
      |                                                                              ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:83:7: note: candidates are: 'std::memory_order std::memory_order_release'
   83 |       memory_order_release,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:206:19: note:                 'const int memory_order_release'
  206 |         const int memory_order_release = 3;
      |                   ^~~~~~~~~~~~~~~~~~~~
io/CompressedInputStream.cpp:761:37: error: reference to 'memory_order_relaxed' is ambiguous
  761 |         if (_processedBlockId->load(memory_order_relaxed) == _blockId - 1)
      |                                     ^~~~~~~~~~~~~~~~~~~~
/home/user/sources/toolchain/arm-gnu-toolchain-12.2.rel1-x86_64-arm-none-eabi/arm-none-eabi/include/c++/12.2.1/bits/atomic_base.h:80:7: note: candidates are: 'std::memory_order std::memory_order_relaxed'
   80 |       memory_order_relaxed,
      |       ^~~~~~~~~~~~~~~~~~~~
io/../concurrent.hpp:204:19: note:                 'const int memory_order_relaxed'
  204 |         const int memory_order_relaxed = 0;
      |                   ^~~~~~~~~~~~~~~~~~~~

Used ARM GNU Toolchain from here (12.2.Rel1 December 22, 2022). If I understand correctly, these errors is about same names in different namespaces used by default simultaneously. What can I do with it? Concerning building in native environment, it works perfectly: the library is three times smaller than the full one. And almost five times smaller when using size optimization instead of speed. It's very great!

[flanglet]

Ah I see the problem. It should be fixed now.

[Ykidia]

Thank you! It builds ok. Now I will think about how to use it as a kernel unpacker)