search

jubako / arx

Store files and directory in an archive. Like tar, but faster and with direct random access.
MIT License
151 stars 2 forks source link

ERROR build on arm 32 bits failed (crash on jubako v0.3.0) #50

Closed father-mande closed 2 months ago

father-mande commented 2 months ago

Hi,

Last git commit ... start generation of arx on an arm 32 bits (Ubuntu 2204) FAIL on size type ... . Compiling jubako v0.3.0-dev (https://github.com/jubako/jubako.git#a2c47ed3) error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/a2c47ed/src/creator/directory_pack/value.rs:18:1 18 sa::assert_eq_size!(Array, [u8; 24]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

= note: source type: creator::directory_pack::value::Array (256 bits) = note: target type: [u8; 24] (192 bits) = note: this error originates in the macro sa::assert_eq_size (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/a2c47ed/src/creator/directory_pack/value.rs:59:1 59 sa::assert_eq_size!(ArrayS<0>, [u8; 16]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

= note: source type: ArrayS<0> (192 bits) = note: target type: [u8; 16] (128 bits) = note: this error originates in the macro sa::assert_eq_size (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/a2c47ed/src/creator/directory_pack/value.rs:61:1 61 sa::assert_eq_size!(ArrayS<1>, [u8; 20]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

= note: source type: ArrayS<1> (192 bits) = note: target type: [u8; 20] (160 bits) = note: this error originates in the macro sa::assert_eq_size (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/a2c47ed/src/creator/directory_pack/value.rs:63:1 63 sa::assert_eq_size!(ArrayS<2>, [u8; 20]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

= note: source type: ArrayS<2> (192 bits) = note: target type: [u8; 20] (160 bits) = note: this error originates in the macro sa::assert_eq_size (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/a2c47ed/src/creator/directory_pack/value.rs:108:1 108 sa::assert_eq_size!(Value, [u8; 12]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
= note: source type: `creator::directory_pack::value::Value` (128 bits)
= note: target type: `[u8; 12]` (96 bits)
= note: this error originates in the macro `sa::assert_eq_size` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try rustc --explain E0512. error: could not compile jubako (lib) due to 5 previous errors warning: build failed, waiting for other jobs to finish... .

Philippe.

mgautierfr commented 2 months ago

Hum, this is strange, I compile in 32 bits in the CI...

Can you run :

RUSTFLAGS=-Zprint-type-sizes cargo +nightly build --no-default-features > type-sizes32.txt

and send me the content of type-size32.txt please ?

father-mande commented 2 months ago

Hi, root@AS1002TAPHIL:/opt/from_git/arx# RUSTFLAGS=-Zprint-type-sizes cargo +nightly build --no-default-features > type-sizes32.txt error: no such command: +nightly

    Cargo does not handle `+toolchain` directives.
    Did you mean to invoke `cargo` through `rustup` instead?

root@AS1002TAPHIL:/opt/from_git/arx#

Philippe

father-mande commented 2 months ago

Hi,

When I check commit in jubako repository it's around the same hour I run the test, do you think it's possible that it's the previous build problem ?

Philippe.

mgautierfr commented 2 months ago

error: no such command: +nightly

Can you install nightly toolchain with rustup toolchain install nightly and re-run command please

do you think it's possible that it's the previous build problem ?

I don't think so

father-mande commented 2 months ago

Hi,

Sorry for the delay, I am a total newbie in rust ... so need time to understand rustup etc. etc. but result

type-sizes32.txt

Philippe.

mgautierfr commented 2 months ago

Hum, this doesn't contain types for jubako. Can you clear first (cargo clean) then rerun the command to be sure it compile (and print type size) for everything ?

father-mande commented 2 months ago

Hi,

Done ... HEREAFTER SCREEN output ... file is joined ...

[Uploading type-sizes32-1.txt…]()

root@AS1002TAPHIL:/opt/from_git/arx# cargo clean Removed 973 files, 355.4MiB total root@AS1002TAPHIL:/opt/from_git/arx# . /root/.cargo/env root@AS1002TAPHIL:/opt/from_git/arx# cargo clean Removed 0 files root@AS1002TAPHIL:/opt/from_git/arx# RUSTFLAGS=-Zprint-type-sizes cargo +nightly build --no-default-features > type-sizes32-1.txt Compiling libc v0.2.158 Compiling shlex v1.3.0 Compiling cfg-if v1.0.0 Compiling proc-macro2 v1.0.86 Compiling version_check v0.9.5 Compiling unicode-ident v1.0.12 Compiling jobserver v0.1.32 Compiling cc v1.1.15 Compiling quote v1.0.37 Compiling syn v2.0.76 Compiling pkg-config v0.3.30 Compiling autocfg v1.3.0 Compiling zstd-sys v2.0.13+zstd.1.5.6 Compiling byteorder v1.5.0 Compiling crossbeam-utils v0.8.20 Compiling typenum v1.17.0 Compiling ahash v0.8.11 Compiling generic-array v0.14.7 Compiling target-lexicon v0.12.16 Compiling memchr v2.7.4 Compiling once_cell v1.19.0 Compiling zerocopy v0.7.35 Compiling utf8parse v0.2.2 Compiling anstyle-parse v0.2.5 Compiling allocator-api2 v0.2.18 Compiling is_terminal_polyfill v1.70.1 Compiling anstyle v1.0.8 Compiling colorchoice v1.0.2 Compiling anstyle-query v1.1.1 Compiling anstream v0.6.15 Compiling hashbrown v0.14.5 Compiling pyo3-build-config v0.20.3 Compiling serde v1.0.209 Compiling rustix v0.38.35 Compiling aho-corasick v1.1.3 Compiling serde_derive v1.0.209 Compiling zstd-safe v6.0.6 Compiling regex-syntax v0.8.4 Compiling linux-raw-sys v0.4.14 Compiling bitflags v2.6.0 Compiling clap_lex v0.7.2 Compiling strsim v0.11.1 Compiling heck v0.5.0 Compiling clap_derive v4.5.13 Compiling clap_builder v4.5.15 Compiling regex-automata v0.4.7 Compiling crypto-common v0.1.6 Compiling crossbeam-epoch v0.9.18 Compiling num-traits v0.2.19 Compiling powerfmt v0.2.0 Compiling log v0.4.22 Compiling rayon-core v1.12.1 Compiling deranged v0.3.11 Compiling crossbeam-deque v0.8.5 Compiling clap v4.5.16 Compiling blake3 v1.5.4 Compiling getrandom v0.2.15 Compiling subtle v2.6.1 Compiling portable-atomic v1.7.0 Compiling uuid v1.10.0 Compiling zstd v0.12.4 Compiling block-buffer v0.10.4 Compiling crossbeam-channel v0.5.13 Compiling bzip2-sys v0.1.11+1.0.8 Compiling fastrand v2.1.1 Compiling equivalent v1.0.1 Compiling arrayref v0.3.8 Compiling arrayvec v0.7.6 Compiling crc-catalog v2.4.0 Compiling iana-time-zone v0.1.60 Compiling constant_time_eq v0.3.1 Compiling either v1.13.0 Compiling adler2 v2.0.0 Compiling rayon v1.10.0 Compiling miniz_oxide v0.8.0 Compiling chrono v0.4.38 Compiling crc v3.2.1 Compiling tempfile v3.12.0 Compiling indexmap v2.4.0 Compiling dropout v0.1.0 Compiling digest v0.10.7 Compiling bstr v1.10.0 Compiling serde_spanned v0.6.7 Compiling toml_datetime v0.6.8 Compiling lru v0.11.1 Compiling fxhash v0.2.1 Compiling backtrace v0.3.73 Compiling memmap2 v0.8.0 Compiling crc32fast v1.4.2 Compiling same-file v1.0.6 Compiling static_assertions v1.1.0 Compiling gimli v0.29.0 Compiling adler v1.0.2 Compiling spmc v0.3.0 Compiling pathdiff v0.2.1 Compiling jubako v0.3.0 (https://github.com/jubako/jubako.git#eabb8086) error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/eabb808/src/creator/directory_pack/value.rs:18:1 18 sa::assert_eq_size!(Array, [u8; 24]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

= note: source type: creator::directory_pack::value::Array (256 bits) = note: target type: [u8; 24] (192 bits) = note: this error originates in the macro sa::assert_eq_size (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/eabb808/src/creator/directory_pack/value.rs:59:1 59 sa::assert_eq_size!(ArrayS<0>, [u8; 16]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

= note: source type: ArrayS<0> (192 bits) = note: target type: [u8; 16] (128 bits) = note: this error originates in the macro sa::assert_eq_size (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/eabb808/src/creator/directory_pack/value.rs:61:1 61 sa::assert_eq_size!(ArrayS<1>, [u8; 20]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

= note: source type: ArrayS<1> (192 bits) = note: target type: [u8; 20] (160 bits) = note: this error originates in the macro sa::assert_eq_size (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/eabb808/src/creator/directory_pack/value.rs:63:1 63 sa::assert_eq_size!(ArrayS<2>, [u8; 20]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

= note: source type: ArrayS<2> (192 bits) = note: target type: [u8; 20] (160 bits) = note: this error originates in the macro sa::assert_eq_size (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types --> /root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/eabb808/src/creator/directory_pack/value.rs:108:1 108 sa::assert_eq_size!(Value, [u8; 12]); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
= note: source type: `creator::directory_pack::value::Value` (128 bits)
= note: target type: `[u8; 12]` (96 bits)
= note: this error originates in the macro `sa::assert_eq_size` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try rustc --explain E0512. error: could not compile jubako (lib) due to 5 previous errors warning: build failed, waiting for other jobs to finish...

Philippe.

father-mande commented 2 months ago

Hi,

Just a stupid question : how do you determine if the system where we generate arx / jubako ... is 32 bits

because on ARM it's not so easy ... no "lm" in features, armvxx in uname, some "solutions" are : just if we analyse libc.so for ex. with file it's an ELF32, or by reverse logic ... if ARM in cpuinfo ... check uname -m ... always aarch64 for 64 bits linux ...

Perhaps some other possibilities with rust (and its tools) but I have no experience and are totally newbie with rust., sometimes an option exist to force architecture (32 or 64) so user can get the responsibility to generate with correct value.

Philippe.

mgautierfr commented 2 months ago

Uploading type-sizes32-1.txt…

Seems their was something wrong with github upload.. Anyway, if rust cannot compile jubako, I suspect it will not print the size of jubako structures. Can you try with https://github.com/jubako/jubako/pull/31 (patching Cargo.toml with branch = "no_size_assert") please ? (Don't forget to clean before)

Just a stupid question : how do you determine if the system where we generate arx / jubako ... is 32 bits

I trust cargo :) As in C, 32 bits or 64 bits is "what we are compiling for".

just if we analyse libc.so for ex. with file it's an ELF32, or by reverse logic ... if ARM in cpuinfo ... check uname -m ... always aarch64 for 64 bits linux ...

You probably have a 64 bits processor, with a 64 bits kernel running 32 bits userland applications. What is important for us is userland, so 32 bits. Most (if not all) 64 bits processors can run 32 bits applications.

father-mande commented 2 months ago

Hi,

I will test A.S.A.P.

for the host it's a pure real armv7l system ... I know it's "old" ... on other arm64 (aarch64) all seems to run as well ... in 64 bits ... . root@AS1002TAPHIL:/opt/from_git/arx# cat /proc/cpuinfo processor : 0 model name : ARMv7 Processor rev 1 (v7l) BogoMIPS : 2125.00 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x4 CPU part : 0xc09 CPU revision : 1

processor : 1 model name : ARMv7 Processor rev 1 (v7l) BogoMIPS : 2125.00 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x4 CPU part : 0xc09 CPU revision : 1

Hardware : Marvell Armada 380/381/382/383/384/385/388 (Device Tree) Revision : 0000 Serial : 0000000000000000 root@AS1002TAPHIL:/opt/from_git/arx# . Philippe.

father-mande commented 2 months ago

Hi,

Compilation OK ... arx is created without error, I will start some tests to verify to don't have difference with "master" branch on other platform.

In advance, lot of thanks. Just a small message, don't break your week-end for my issues ... I am an independent integrator (so not really a developer) of applications on NAS by creating package when I think a software can be interesting for NAS user and usage ... so I have some NAS models on different architecture to run tests (so never standard distribution) .

I work on it, on my free time only (so sometimes also week-end) ... so keep your priority ... and keep your free time ...

Regards. Philippe.

father-mande commented 2 months ago

Hi,

Last test : arx build OK arx create (5GB) ... OK arx list CRASH

"name" = "arx" "operating_system" = "Linux [unknown bitness]" "crate_version" = "0.3.0" "explanation" = """ Panic occurred in file '/root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/f685762/src/bases/io/file.rs' at line 97 """ "cause" = "not implemented: Block of not memory block is not implemented" "method" = "Panic" "backtrace" = """

0: 0xb6b31508 - core::panicking::panic_fmt::he0ea9e2f8e691d3e 1: 0xb6cc8b90 - ::cut::h63640d2f569b8d17 2: 0xb6c9bce8 - ::finalize::h351ba4a569558cb6 3: 0xb6c8f064 - jubako::bases::reader::Reader::parse_data_block::hd7b9ab423fb362b7 4: 0xb6c9cfd8 - <jubako::reader::directory_pack::DirectoryPack as jubako::bases::cache::CachableSource>::get_value::h4e6311fc4acd4cc4 5: 0xb6c9c054 - jubako::reader::directory_pack::EntryStorage::get_entry_store::h6943f2f9590f7b41 6: 0xb6cc13b4 - jubako::reader::directory_pack::index::Index::get_store::h5a8cf3e8c8c85de1 7: 0xb6c67c94 - libarx::arx::create_properties::hed7b08981e98f325 8: 0xb6b81b5c - libarx::arx::Arx::new::h35c7cb595673614e 9: 0xb6b472b4 - arx::list::list::h48d80768ce497a3c 10: 0xb6b38210 - arx::run::h2a8d65c3a41409c9 11: 0xb6b39ab0 - arx::main::he9ee27ab2ac87f91 12: 0xb6b521a4 - std::sys_common::backtrace::rust_begin_short_backtrace::h4d5527f5d6b46f03 13: 0xb6b8aa6c - std::rt::lang_start::{{closure}}::h4c8d5855fac6e145 14: 0xb6e61188 - std::rt::lang_start_internal::ha584fb11ead9091f 15: 0xb6b3eb7c - main 16: 0xb68fd7d6 - 17: 0xb68fd886 - libc_start_main"""

arx mount CRASH

"name" = "arx" "operating_system" = "Linux [unknown bitness]" "crate_version" = "0.3.0" "explanation" = """ Panic occurred in file '/root/.cargo/git/checkouts/jubako-d6d555d5cd7df63d/f685762/src/bases/io/file.rs' at line 97 """ "cause" = "not implemented: Block of not memory block is not implemented" "method" = "Panic" "backtrace" = """

0: 0xb6b55508 - core::panicking::panic_fmt::he0ea9e2f8e691d3e 1: 0xb6cecb90 - ::cut::h63640d2f569b8d17 2: 0xb6cbfce8 - ::finalize::h351ba4a569558cb6 3: 0xb6cb3064 - jubako::bases::reader::Reader::parse_data_block::hd7b9ab423fb362b7 4: 0xb6cc0fd8 - <jubako::reader::directory_pack::DirectoryPack as jubako::bases::cache::CachableSource>::get_value::h4e6311fc4acd4cc4 5: 0xb6cc0054 - jubako::reader::directory_pack::EntryStorage::get_entry_store::h6943f2f9590f7b41 6: 0xb6ce53b4 - jubako::reader::directory_pack::index::Index::get_store::h5a8cf3e8c8c85de1 7: 0xb6c8bc94 - libarx::arx::create_properties::hed7b08981e98f325 8: 0xb6ba5b5c - libarx::arx::Arx::new::h35c7cb595673614e 9: 0xb6ba7e7c - arx::mount::mount::h9a7571138bd53208 10: 0xb6b5c0f8 - arx::run::h2a8d65c3a41409c9 11: 0xb6b5dab0 - arx::main::he9ee27ab2ac87f91 12: 0xb6b761a4 - std::sys_common::backtrace::rust_begin_short_backtrace::h4d5527f5d6b46f03 13: 0xb6baea6c - std::rt::lang_start::{{closure}}::h4c8d5855fac6e145 14: 0xb6e85188 - std::rt::lang_start_internal::ha584fb11ead9091f 15: 0xb6b62b7c - main 16: 0xb691d7d6 - 17: 0xb691d886 - libc_start_main"""

Philippe

mgautierfr commented 2 months ago

Can you try with no_fail_block_check branch (https://github.com/jubako/jubako/pull/32) of Jubako please.

Even with https://github.com/jubako/jubako/pull/31 which fix the compilation issue, I'm still interested in the size of types to understand what happening. If you have time, no urgency.

Don't worry of my free time. I'm also creating arx on my spare time and I'm please to have a feedback so I want to help. And I will have no time at all (or almost) very soon for few weeks, so I prefer few hours spend now and "leave" with something working than let you blocked.

father-mande commented 2 months ago

Hi,

Test with no_fail_block_check branch

... arx binary creation : OK

TEST with a directory with 764 Mbytes in ... arx create OK ... arx list OK ... arx mount OK

TEST with a directory with 5 GB GigaBytes in ... arx create fail ... work with previous version

EDIT EDIT EDIT ... error is mine, I work in parallel on other project and try to archive a running chroot ... SO SORRY ... rerun tests ...

arx create is running ... I will wait

father-mande commented 2 months ago

Hi,

I edit the previous message due my (stupid) error ...

Even a question exist about is : it mandatory to don't have task using a folder where we archive ? or is it specific to chroot and mount active inside ...

Philippe.

father-mande commented 2 months ago

Hi,

Last test on large folder 5GB arx create OK arx list OK arx mount OK

Thanks ... Best regards. Philippe

mgautierfr commented 2 months ago

Even a question exist about is : it mandatory to don't have task using a folder where we archive ? or is it specific to chroot and mount active inside ...

It is not mandatory but highly recommended. This is not arx specific. Even tar has the same issue, if you archive/backup a directory as you are modifying it, you don't know what exactly you are saving.

One solution is something like btrfs (or equivalent) snapshot which atomically create "copy" of a directory.

father-mande commented 2 months ago

Hi,

TRUE (sorry, I normally know this) ... I will add a suggestion for NAS users to use lsof for ex. to check before starting archive ...

I think if you can merge in jubako ... if modification don't break for other architecture ... you can closed this issue.

Again lot of thanks. Philippe.