error initializing file system: unsupported compression type: ZSTD

Turkceqwertyx commented 9 months ago

Hello, i am using Gentoo with openrc. my dwarfs version is dwarfs-0.7.2-r1. i am using btrfs with compress=zstd:3as my filesystem and this error happens when i am trying to open a dwarfs archive. My use flags is here:

+ + jemalloc                        : Use dev-libs/jemalloc for memory management
 - - man                             : Build and install man pages
 - - python                          : Add optional support/bindings for the Python language
 - - python_single_target_python3_10 : Build for Python 3.10 only
 + + python_single_target_python3_11 : Build for Python 3.11 only
 - - python_single_target_python3_12 : Build for Python 3.12 only
 - - test                            : Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)

++ is on and -- is off

this is my fstab:

/dev/nvme0n1p2 / btrfs compress=zstd:3,ssd,noatime,defaults 0 1

here is the output of dwarfsextract -i groot.dwarfs -o groot/ -l debug

D 03:38:17 [filesystem_v2.cpp:445] found valid section index D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 64 [18.098.817 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 18.098.945 [21.489.586 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 39.588.595 [21.839.671 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 61.428.330 [32.151.923 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 93.580.317 [33.776.695 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 127.357.076 [16.102.117 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 143.459.257 [20.759.304 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 164.218.625 [32.182.074 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 196.400.763 [25.387.782 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 221.788.609 [26.936.929 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 248.725.602 [21.449.577 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 270.175.243 [24.178.224 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 294.353.531 [21.531.328 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 315.884.923 [33.314.286 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 349.199.273 [26.665.109 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 375.864.446 [26.460.134 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 402.324.644 [25.337.940 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 427.662.648 [23.208.872 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 450.871.584 [23.928.441 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 474.800.089 [20.071.887 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 494.872.040 [32.698.013 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 527.570.117 [26.305.097 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 553.875.278 [29.321.114 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 583.196.456 [29.500.323 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 612.696.843 [27.441.925 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 640.138.832 [37.435.211 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 677.574.107 [27.589.496 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 705.163.667 [31.649.684 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 736.813.415 [32.118.236 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 768.931.715 [24.762.166 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 793.693.945 [20.303.087 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 813.997.096 [20.626.088 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 834.623.248 [39.374.582 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 873.997.894 [24.643.071 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 898.641.029 [18.318.889 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 916.959.982 [24.422.221 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 941.382.267 [29.157.265 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 970.539.596 [39.511.716 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.010.051.376 [14.728.065 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.024.779.505 [19.560.559 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.044.340.128 [33.339.449 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.077.679.641 [27.306.846 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.104.986.551 [26.670.107 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.131.656.722 [24.142.656 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.155.799.442 [21.378.577 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.177.178.083 [26.786.190 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.203.964.337 [30.094.439 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.234.058.840 [27.334.712 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.261.393.616 [24.815.646 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.286.209.326 [23.540.851 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.309.750.241 [27.390.119 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.337.140.424 [18.081.543 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.355.222.031 [36.391.946 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.391.614.041 [21.794.368 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.413.408.473 [30.382.297 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.443.790.834 [32.366.384 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.476.157.282 [28.294.234 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.504.451.580 [36.654.911 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.541.106.555 [31.901.117 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.573.007.736 [32.229.431 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.605.237.231 [31.598.496 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.636.835.791 [27.397.261 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.664.233.116 [19.584.098 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.683.817.278 [20.436.006 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.704.253.348 [40.129.076 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.744.382.488 [41.279.178 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.785.661.730 [50.024.692 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.835.686.486 [22.620.219 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.858.306.769 [22.078.570 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.880.385.403 [19.215.622 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.899.601.089 [17.199.805 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.916.800.958 [16.926.940 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.933.727.962 [46.034.374 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 1.979.762.400 [58.866.729 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.038.629.193 [57.269.379 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.095.898.636 [63.156.952 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.159.055.652 [63.372.792 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.222.428.508 [65.632.891 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.288.061.463 [65.597.232 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.353.658.759 [65.630.999 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.419.289.822 [65.370.344 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.484.660.230 [63.316.361 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.547.976.655 [50.639.754 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.598.616.473 [57.799.637 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.656.416.174 [58.253.242 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.714.669.480 [60.028.046 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.774.697.590 [41.365.459 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.816.063.113 [29.140.599 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.845.203.776 [59.457.592 bytes] D 03:38:17 [filesystem_v2.cpp:453] section BLOCK @ 2.904.661.432 [16.015.711 bytes] D 03:38:17 [filesystem_v2.cpp:453] section METADATA_V2_SCHEMA @ 2.920.677.207 [447 bytes] D 03:38:17 [filesystem_v2.cpp:453] section METADATA_V2 @ 2.920.677.718 [223.226 bytes] D 03:38:17 [filesystem_v2.cpp:453] section SECTION_INDEX @ 2.920.901.008 [744 bytes] D 03:38:17 [block_cache.cpp:156] stopping cache workers dwarfs::runtime_error: unsupported compression type: ZSTD

mhx commented 9 months ago

Hi, thanks for your report.

It unfortunately looks like nobody has really tested the dwarfs ebuild properly and it is badly broken. Looking at the ebuild itself, it seems to be building the internal libraries, in particular libdwarfs_compression, as shared libraries, which is not supported. This causes the individual tools, including dwarfsextract, to not find any compression implementations.

Please file a bug report for the dwarfs ebuild.

In the meantime, you can download a binary release or a universal binary.

zhuyifei1999 commented 9 months ago

I had a quick look at this (because reddit) and I couldn't figure out how the ebuild should specify these libraries to make libdwarfs_compression become static libraries (building everything static seems to be highly discouraged and I'm not sure how to toggle each library individually).

Then I took a look at the actual compilation commands on my side. This is that of dwarfsextract:

x86_64-pc-linux-gnu-g++ -march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=12 -fuse-linker-plugin -falign-functions=32 -pipe -ggdb -frecord-gcc-switches -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -I/usr/include -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=12 -fuse-linker-plugin -falign-functions=32 -pipe -ggdb -frecord-gcc-switches CMakeFiles/dwarfsextract.dir/src/dwarfsextract.cpp.o -o dwarfsextract  -Wl,-rpath,/var/tmp/portage/sys-fs/dwarfs-0.7.4/work/dwarfs-0.7.4:/var/tmp/portage/sys-fs/dwarfs-0.7.4/work/dwarfs-0.7.4/folly:  libdwarfsextract_main.so  -Wl,--push-state,--whole-archive  libdwarfs_compression.so  -Wl,--pop-state  libdwarfs_tool.so  libdwarfs.so  libmetadata_thrift.so  libthrift_light.so  libfsst.so  /usr/lib64/libboost_chrono.so.1.83.0  /usr/lib64/libboost_iostreams.so.1.83.0  /usr/lib64/libbrotlidec.so  /usr/lib64/libbrotlienc.so  /usr/lib64/libarchive.so  /usr/lib64/libzstd.so  /usr/lib64/libxxhash.so  /usr/lib64/libjemalloc.so  folly/libfolly.so.0.58.0-dev  /usr/lib64/libboost_program_options.so.1.83.0  /usr/lib64/libfmt.so.9.1.0  /usr/lib64/libboost_context.so.1.83.0  /usr/lib64/libboost_filesystem.so.1.83.0  /usr/lib64/libboost_atomic.so.1.83.0  /usr/lib64/libboost_regex.so.1.83.0  /usr/lib64/libboost_system.so.1.83.0  /usr/lib64/libboost_thread.so.1.83.0  /usr/lib64/libdouble-conversion.so  /usr/lib64/libgflags.so.2.2.2  /usr/lib64/libglog.so  /usr/lib64/libevent.so  /usr/lib64/libz.so  /usr/lib64/libssl.so  /usr/lib64/libcrypto.so  /usr/lib64/libbz2.so  /usr/lib64/liblz4.so  /usr/lib64/liblzma.so  /usr/lib64/libsnappy.so  /usr/lib64/libiberty.a  /usr/lib64/libaio.so  /usr/lib64/liburing.so  /usr/lib64/libsodium.so  -ldl  /usr/lib64/libunwind.so

Despite having -Wl,--push-state,--whole-archive libdwarfs_compression.so -Wl,--pop-state, libdwarfs_compression.so does not appear in ldd dwarfsextract.

This is a case of -Wl,--as-needed doing its thing. -Wl,--as-needed is part of my system CFLAGS, so it deletes the DT_NEEDED entry for libdwarfs_compression.so since it thinks it's not necessary. Removing that flag, or changing -Wl,--push-state,--whole-archive to -Wl,--push-state,--whole-archive,--no-as-needed restores the SO in the ldd output.

mhx commented 9 months ago

Thanks for the additional information, @zhuyifei1999!

Indeed, --as-needed is a culprit here. The following patch fixes the ebuild for me:

--- dwarfs-0.7.2-r1.ebuild.orig 2024-01-05 01:19:54.486584449 +0100
+++ dwarfs-0.7.2-r1.ebuild  2024-01-05 01:13:57.399919972 +0100
@@ -80,6 +80,7 @@

 src_configure(){
    append-cxxflags "-I/usr/include"
+   append-ldflags $(no-as-needed)

    mycmakeargs=(
        -DUSE_JEMALLOC=$(usex jemalloc ON OFF)
@@ -111,6 +112,7 @@
    cmake_src_install
    dolib.so libdwarfs.so libdwarfs_main.so libdwarfs_tool.so libdwarfs_compression.so libthrift_light.so libmetadata_thrift.so || die "Install failed"
    dolib.so folly/libfolly.so folly/libfolly.so.0.58.0-dev libmkdwarfs_main.so libdwarfsbench_main.so libdwarfsck_main.so libdwarfsextract_main.so || die "Install failed"
+   dolib.so folly/folly/experimental/exception_tracer/libfolly_exception_tracer_base.so folly/folly/experimental/exception_tracer/libfolly_exception_tracer_base.so.0.58.0-dev || die "Install failed"
 }

 pkg_postinst(){

Pinging @RarogCmex, as it seems they're the author of the ebuild.

RarogCmex commented 8 months ago

@mhx That's the weirdest, that I have pure 64bit system and ebuild works! And I have thought shared libs were intended. Furthermore there is no need in experimental system. !!! dolib: folly/folly/experimental/exception_tracer/libfolly_exception_tracer_base.so does not exist So as @zhuyifei1999 mentioned --as-needed brokes everything.

I will update ebuild with append-ldflags $(no-as-needed) To be honest, I have no more performed active gentoo maintainership since I became undergraduate student of Business Informatics in financial university under the government of the Russian Federation. That long text states that I am a bit ovewhelmed, so I may stop maintaining ebuild completely.

RarogCmex commented 8 months ago

@zhuyifei1999 Also there are sys-fs/dwarfs-bin in the GURU tree, which installs prebuilt version and therefore it is a lot more reliable. I will update it too.

zhuyifei1999 commented 8 months ago

To be honest, I have no more performed active gentoo maintainership

@RarogCmex if you don't mind I can co-maintain these packages in GURU. I maintain a few packages of my own there (I am also a GURU contributer)

RarogCmex commented 8 months ago

@zhuyifei1999 I would be happy. Just add your maintainership in metadata.xml.

Turkceqwertyx commented 8 months ago

i am closing this issue because it's fixed. thank you guys!

mhx / dwarfs

error initializing file system: unsupported compression type: ZSTD #184