search

Gottox / sqsh-tools

🗜️ fast r/o squashfs implementation written in C.
https://gottox.de/sqsh-tools/
BSD 2-Clause "Simplified" License
37 stars 4 forks source link

Fails to build on macos #77

Closed Dr-Emann closed 1 year ago

Dr-Emann commented 1 year ago

Describe the bug

Build fails on macos because #include <endian.h>

To Reproduce

meson setup \
  -Ddefault_library=static \
  -Dexamples=true \
  -Db_lundef=false \
  -Dtest=false \
  -Ddoc=true \
  -Dfuzzer_timeout=10 \
  -Dcurl=enabled \
  -Dzlib=enabled \
  -Dlz4=enabled \
  -Dlzma=enabled \
  -Dzstd=enabled \
  -Db_coverage=true \
  -Db_sanitize=address,undefined \
  -Dwerror=true \
  "build" && ninja -C build all
Full output ``` The Meson build system Version: 1.2.1 Source dir: /Users/zach/Development/tmp/libsqsh Build dir: /Users/zach/Development/tmp/libsqsh/build Build type: native build Project name: libsqsh Project version: 1.0.0 C compiler for the host machine: sccache cc (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)") C linker for the host machine: cc ld64 857.1 Host machine cpu family: aarch64 Host machine cpu: aarch64 Run-time dependency threads found: YES Found pkg-config: /opt/homebrew/bin/pkg-config (0.29.2) Run-time dependency libcurl found: YES 7.79.1 Found CMake: /opt/homebrew/bin/cmake (3.27.4) Run-time dependency fuse3 found: NO (tried pkgconfig, framework and cmake) Dependency fuse skipped: feature fuse-old disabled Run-time dependency liblz4 found: YES 1.9.4 Run-time dependency liblzma found: YES 5.4.4 Run-time dependency zlib found: YES 1.2.11 Run-time dependency libzstd found: YES 1.5.5 Executing subproject cextras cextras| Project name: cextras cextras| Project version: 0.1 cextras| C compiler for the host machine: sccache cc (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)") cextras| C linker for the host machine: cc ld64 857.1 cextras| Dependency threads found: YES unknown (cached) cextras| Build targets in project: 2 cextras| Subproject cextras finished. Configuring sqsh-cat.1 using configuration Configuring sqsh-ls.1 using configuration Configuring sqsh-stat.1 using configuration Configuring sqsh-unpack.1 using configuration Configuring sqsh-xattr.1 using configuration Program doxygen found: YES (/opt/homebrew/bin/doxygen) Program git found: YES (/opt/homebrew/bin/git) Configuring Doxyfile using configuration Build targets in project: 15 libsqsh 1.0.0 Subprojects cextras : YES User defined options default_library: static werror : true b_coverage : true b_lundef : false b_sanitize : address,undefined curl : enabled doc : true examples : true fuzzer_timeout : 10 lz4 : enabled lzma : enabled test : false zlib : enabled zstd : enabled Found ninja-1.11.1 at /opt/homebrew/bin/ninja WARNING: Need gcovr or lcov/genhtml to generate any coverage reports ninja: Entering directory `build' [1/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/collection_buffer.c.o [2/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/collection_rc_hash_map.c.o [3/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/concurrency_future.c.o [4/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/collection_lru.c.o [5/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/collection_rc_map.c.o [6/84] Compiling C object lib/libsqsh.1.dylib.p/reader_reader.c.o [7/84] Compiling C object lib/libsqsh.1.dylib.p/archive_trailing_context.c.o [8/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/concurrency_threadpool.c.o [9/84] Compiling C object lib/libsqsh.1.dylib.p/archive_archive.c.o [10/84] Compiling C object lib/libsqsh.1.dylib.p/archive_compression_options.c.o [11/84] Linking static target subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a [12/84] Compiling C object lib/libsqsh.1.dylib.p/archive_superblock.c.o [13/84] Compiling C object lib/libsqsh.1.dylib.p/archive_inode_map.c.o [14/84] Compiling C object lib/libsqsh.1.dylib.p/data_directory_data.c.o FAILED: lib/libsqsh.1.dylib.p/data_directory_data.c.o sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_directory_data.c.o -MF lib/libsqsh.1.dylib.p/data_directory_data.c.o.d -o lib/libsqsh.1.dylib.p/data_directory_data.c.o -c ../lib/data/directory_data.c ../lib/data/directory_data.c:41:11: fatal error: 'endian.h' file not found # include ^~~~~~~~~~ 1 error generated. [15/84] Compiling C object lib/libsqsh.1.dylib.p/data_fragment_data.c.o FAILED: lib/libsqsh.1.dylib.p/data_fragment_data.c.o sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_fragment_data.c.o -MF lib/libsqsh.1.dylib.p/data_fragment_data.c.o.d -o lib/libsqsh.1.dylib.p/data_fragment_data.c.o -c ../lib/data/fragment_data.c ../lib/data/fragment_data.c:41:11: fatal error: 'endian.h' file not found # include ^~~~~~~~~~ 1 error generated. [16/84] Compiling C object lib/libsqsh.1.dylib.p/data_compression_options_data.c.o FAILED: lib/libsqsh.1.dylib.p/data_compression_options_data.c.o sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_compression_options_data.c.o -MF lib/libsqsh.1.dylib.p/data_compression_options_data.c.o.d -o lib/libsqsh.1.dylib.p/data_compression_options_data.c.o -c ../lib/data/compression_options_data.c ../lib/data/compression_options_data.c:41:11: fatal error: 'endian.h' file not found # include ^~~~~~~~~~ 1 error generated. [17/84] Compiling C object lib/libsqsh.1.dylib.p/data_metablock_data.c.o FAILED: lib/libsqsh.1.dylib.p/data_metablock_data.c.o sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_metablock_data.c.o -MF lib/libsqsh.1.dylib.p/data_metablock_data.c.o.d -o lib/libsqsh.1.dylib.p/data_metablock_data.c.o -c ../lib/data/metablock_data.c ../lib/data/metablock_data.c:41:11: fatal error: 'endian.h' file not found # include ^~~~~~~~~~ 1 error generated. [18/84] Compiling C object lib/libsqsh.1.dylib.p/data_superblock_data.c.o FAILED: lib/libsqsh.1.dylib.p/data_superblock_data.c.o sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_superblock_data.c.o -MF lib/libsqsh.1.dylib.p/data_superblock_data.c.o.d -o lib/libsqsh.1.dylib.p/data_superblock_data.c.o -c ../lib/data/superblock_data.c ../lib/data/superblock_data.c:41:11: fatal error: 'endian.h' file not found # include ^~~~~~~~~~ 1 error generated. [19/84] Compiling C object lib/libsqsh.1.dylib.p/data_inode_data.c.o FAILED: lib/libsqsh.1.dylib.p/data_inode_data.c.o sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_inode_data.c.o -MF lib/libsqsh.1.dylib.p/data_inode_data.c.o.d -o lib/libsqsh.1.dylib.p/data_inode_data.c.o -c ../lib/data/inode_data.c ../lib/data/inode_data.c:41:11: fatal error: 'endian.h' file not found # include ^~~~~~~~~~ 1 error generated. [20/84] Compiling C object lib/libsqsh.1.dylib.p/data_xattr_data.c.o FAILED: lib/libsqsh.1.dylib.p/data_xattr_data.c.o sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_xattr_data.c.o -MF lib/libsqsh.1.dylib.p/data_xattr_data.c.o.d -o lib/libsqsh.1.dylib.p/data_xattr_data.c.o -c ../lib/data/xattr_data.c ../lib/data/xattr_data.c:41:11: fatal error: 'endian.h' file not found # include ^~~~~~~~~~ 1 error generated. [21/84] Compiling C object lib/libsqsh.1.dylib.p/directory_directory_index_iterator.c.o [22/84] Compiling C object lib/libsqsh.1.dylib.p/directory_directory_iterator.c.o [23/84] Compiling C object lib/libsqsh.1.dylib.p/easy_directory.c.o [24/84] Compiling C object lib/libsqsh.1.dylib.p/extract_extract_manager.c.o [25/84] Compiling C object lib/libsqsh.1.dylib.p/easy_file.c.o ninja: build stopped: subcommand failed. ```

Expected behavior

Build successful

Platform (please complete the following information):

uname -a output:

Darwin zach-mbp.lan 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000 arm64

Additional context I was able to hackily limp along by removing the includes, and defining no-ops for le32toh (and the like), and it seemed to succeed, so it seems like it's close to supporting macos.

Gottox commented 1 year ago

MacOS again... 🤬

JSchlarb commented 1 year ago

grafik

Local build output 
➜  libsqsh git:(origin/fix/macos-build) meson test -C /tmp/build -t 5
ninja: Entering directory `/private/tmp/build'
[169/169] Linking target test/read-chunk
 1/38 archive/compression_options.c                       OK               0.17s
 2/38 easy/directory.c                                    OK               0.33s
 3/38 easy/file.c                                         OK               0.49s
 4/38 cpp-test.cpp                                        OK               0.64s
 5/38 directory/directory_iterator.c                      OK               0.81s
 6/38 extract/extract.c                                   OK               0.98s
 7/38 extract/extract_manager.c                           OK               1.10s
 8/38 file/file_iterator.c                                OK               1.26s
 9/38 file/file_reader.c                                  OK               1.38s
10/38 fuzzer_repro.c                                      OK               1.57s
11/38 include_tests/sqsh_archive.c                        OK               1.55s
12/38 include_tests/sqsh_easy.c                           OK               1.53s
13/38 include_tests/sqsh_common.c                         OK               1.52s
14/38 include_tests/sqsh_directory.c                      OK               1.57s
15/38 include_tests/sqsh_error.c                          OK               1.56s
16/38 include_tests/sqsh_file.c                           OK               1.56s
17/38 include_tests/sqsh_mapper.c                         OK               1.58s
18/38 include_tests/sqsh_table.c                          OK               1.59s
19/38 include_tests/sqsh_tree.c                           OK               1.61s
20/38 include_tests/sqsh_xattr.c                          OK               1.57s
21/38 file/file.c                                         OK               1.60s
22/38 integration.c                                       OK               1.67s
23/38 mapper/map_iterator.c                               OK               1.59s
24/38 mapper/map_reader.c                                 OK               1.55s
25/38 metablock/metablock_iterator.c                      OK               1.54s
26/38 metablock/metablock_reader.c                        OK               1.54s
27/38 nasty.c                                             OK               1.51s
28/38 reader/reader.c                                     OK               1.49s
29/38 tree/walker.c                                       OK               1.53s
30/38 xattr/xattr_iterator.c                              OK               1.47s
31/38 tools/unpack/pathtraversal/pathtraversal.sh         OK               2.07s
32/38 tools/unpack/repack.sh                              OK               2.39s
33/38 tools/read-chunk/tail.sh                            OK               2.69s
34/38 selftest.sh                                         OK               4.90s
35/38 tools/cat/large-file-compressed.sh                  OK               6.56s
36/38 tools/cat/large-file-uncompressed.sh                OK               6.68s
37/38 tools/fs/large-file.sh (sqshfs2)                    FAIL            52.20s   exit status 1
>>> SOURCE_ROOT=/private/tmp/libsqsh SQSH_CAT=/private/tmp/build/tools/sqsh-cat SQSH_UNPACK=/private/tmp/build/tools/sqsh-unpack MALLOC_PERTURB_=120 BUILD_DIR=/private/tmp/build/test SQSH_LS=/private/tmp/build/tools/sqsh-ls SQSHFS=/private/tmp/build/tools/sqshfs2 MKSQUASHFS=/opt/homebrew/bin/mksquashfs READ_CHUNK=/private/tmp/build/test/read-chunk /private/tmp/libsqsh/test/tools/fs/large-file.sh

38/38 tools/fs/repack.sh (sqshfs2)                        FAIL           110.91s   exit status 1
>>> SOURCE_ROOT=/private/tmp/libsqsh SQSH_CAT=/private/tmp/build/tools/sqsh-cat SQSH_UNPACK=/private/tmp/build/tools/sqsh-unpack MALLOC_PERTURB_=249 BUILD_DIR=/private/tmp/build/test SQSH_LS=/private/tmp/build/tools/sqsh-ls SQSHFS=/private/tmp/build/tools/sqshfs2 MKSQUASHFS=/opt/homebrew/bin/mksquashfs READ_CHUNK=/private/tmp/build/test/read-chunk /private/tmp/libsqsh/test/tools/fs/repack.sh

Ok:                 36
Expected Fail:      0
Fail:               2
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /private/tmp/build/meson-logs/testlog.txt

➜  libsqsh git:(origin/fix/macos-build) sw_vers
ProductName:        macOS
ProductVersion:     13.5
BuildVersion:       22G74
➜  libsqsh git:(origin/fix/macos-build) uname -a
Darwin blub.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm
Gottox commented 1 year ago

Automated testing of fuse filesystems on macos seems to be hard nowadays:

https://github.com/s3fs-fuse/s3fs-fuse/blob/e8cb6d6d3487507f5af62ac555a5dc21556f4571/.github/workflows/ci.yml#L126-L133

JSchlarb commented 1 year ago

Tests passed for 09d551911e89a66a5b44b4a806f0b047ebec722c

Local build output 
➜  libsqsh git:(origin/fix/macos-build) meson test -C /tmp/build -t 5
ninja: Entering directory `/private/tmp/build'
[165/165] Linking target tools/sqsh-cat
 1/36 archive/compression_options.c                       OK               0.16s
 2/36 easy/directory.c                                    OK               0.32s
 3/36 easy/file.c                                         OK               0.46s
 4/36 cpp-test.cpp                                        OK               0.60s
 5/36 directory/directory_iterator.c                      OK               0.75s
 6/36 extract/extract.c                                   OK               0.90s
 7/36 extract/extract_manager.c                           OK               1.04s
 8/36 file/file_iterator.c                                OK               1.19s
 9/36 file/file_reader.c                                  OK               1.33s
10/36 fuzzer_repro.c                                      OK               1.50s
11/36 include_tests/sqsh_archive.c                        OK               1.49s
12/36 include_tests/sqsh_easy.c                           OK               1.47s
13/36 include_tests/sqsh_common.c                         OK               1.48s
14/36 include_tests/sqsh_directory.c                      OK               1.47s
15/36 include_tests/sqsh_error.c                          OK               1.47s
16/36 include_tests/sqsh_file.c                           OK               1.47s
17/36 include_tests/sqsh_mapper.c                         OK               1.48s
18/36 include_tests/sqsh_table.c                          OK               1.47s
19/36 include_tests/sqsh_tree.c                           OK               1.48s
20/36 include_tests/sqsh_xattr.c                          OK               1.47s
21/36 file/file.c                                         OK               1.49s
22/36 integration.c                                       OK               1.53s
23/36 mapper/map_iterator.c                               OK               1.48s
24/36 mapper/map_reader.c                                 OK               1.48s
25/36 metablock/metablock_iterator.c                      OK               1.49s
26/36 metablock/metablock_reader.c                        OK               1.52s
27/36 nasty.c                                             OK               1.74s
28/36 reader/reader.c                                     OK               1.75s
29/36 tree/walker.c                                       OK               1.77s
30/36 xattr/xattr_iterator.c                              OK               1.75s
31/36 tools/unpack/pathtraversal/pathtraversal.sh         OK               2.04s
32/36 tools/unpack/repack.sh                              OK               2.38s
33/36 tools/read-chunk/tail.sh                            OK               2.76s
34/36 selftest.sh                                         OK               4.67s
35/36 tools/cat/large-file-compressed.sh                  OK               6.84s
36/36 tools/cat/large-file-uncompressed.sh                OK               6.83s

Ok:                 36
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /private/tmp/build/meson-logs/testlog.txt
Gottox commented 1 year ago

fixed in #78

Dr-Emann commented 1 year ago

Thank you!