search

dropbox / rust-brotli

Brotli compressor and decompressor written in rust that optionally avoids the stdlib
https://dropbox.tech/infrastructure/-broccoli--syncing-faster-by-syncing-less
BSD 3-Clause "New" or "Revised" License
818 stars 83 forks source link

Few tests fail with OOM on 32bit #42

Closed ignatenkobrain closed 1 month ago

ignatenkobrain commented 4 years ago 
failures:
---- enc::test::test_roundtrip_compressed_repeated stdout ----
thread 'enc::test::test_roundtrip_compressed_repeated' panicked at 'OOM', /usr/share/cargo/registry/alloc-no-stdlib-2.0.1/src/stack_allocator.rs:52:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
---- enc::test::test_roundtrip_ukkonooa stdout ----
thread 'enc::test::test_roundtrip_ukkonooa' panicked at 'OOM', /usr/share/cargo/registry/alloc-no-stdlib-2.0.1/src/stack_allocator.rs:52:13
failures:
    enc::test::test_roundtrip_compressed_repeated
    enc::test::test_roundtrip_ukkonooa
ignatenkobrain commented 4 years ago

I see taht on armv7hl there are more tests which OOM:

failures:
---- enc::test::test_roundtrip_monkey stdout ----
thread 'enc::test::test_roundtrip_monkey' panicked at 'OOM', /usr/share/cargo/registry/alloc-no-stdlib-2.0.1/src/stack_allocator.rs:52:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
---- enc::test::test_roundtrip_quickfox stdout ----
thread 'enc::test::test_roundtrip_quickfox' panicked at 'OOM', /usr/share/cargo/registry/alloc-no-stdlib-2.0.1/src/stack_allocator.rs:52:13
---- enc::test::test_roundtrip_quickfox_repeated stdout ----
thread 'enc::test::test_roundtrip_quickfox_repeated' panicked at 'OOM', /usr/share/cargo/registry/alloc-no-stdlib-2.0.1/src/stack_allocator.rs:52:13
failures:
    enc::test::test_roundtrip_monkey
    enc::test::test_roundtrip_quickfox
    enc::test::test_roundtrip_quickfox_repeated
danielrh commented 4 years ago

Interesting--all these tests allocate the working-set data on the stack. Is it possible to make them to skip on arm? I doubt you'd choose a stack allocator on Arm, but it was a nice cross platform way to test the nostd 3rd party allocators.

alexanderkjall commented 1 year ago

These test errors also affects the debian builds: https://ci.debian.net/data/autopkgtest/testing/armel/r/rust-brotli/38555788/log.gz

musicinmybrain commented 1 month ago

This can be reproduced in a 32-bit x86 chroot today, with current master f41a05988c1964b799f3b47b8820ed1276647bb2:

# git clone https://github.com/dropbox/rust-brotli.git
# cd rust-brotli
# cargo test
[…]
failures:

---- enc::test::test_roundtrip_64x stdout ----
thread 'enc::test::test_roundtrip_64x' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- enc::test::test_roundtrip_10x10y stdout ----
thread 'enc::test::test_roundtrip_10x10y' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_monkey stdout ----
thread 'enc::test::test_roundtrip_monkey' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_x stdout ----
thread 'enc::test::test_roundtrip_x' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_quickfox stdout ----
thread 'enc::test::test_roundtrip_quickfox' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_ukkonooa stdout ----
thread 'enc::test::test_roundtrip_ukkonooa' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_quickfox_repeated stdout ----
thread 'enc::test::test_roundtrip_quickfox_repeated' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_backward65536 stdout ----
thread 'enc::test::test_roundtrip_backward65536' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_compressed_repeated stdout ----
thread 'enc::test::test_roundtrip_compressed_repeated' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_asyoulik stdout ----
thread 'enc::test::test_roundtrip_asyoulik' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

---- enc::test::test_roundtrip_asyoulik9_5 stdout ----
thread 'enc::test::test_roundtrip_asyoulik9_5' panicked at /builddir/.cargo/registry/src/index.crates.io-1cd66030c949c28d/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM

failures:
    enc::test::test_roundtrip_10x10y
    enc::test::test_roundtrip_64x
    enc::test::test_roundtrip_asyoulik
    enc::test::test_roundtrip_asyoulik9_5
    enc::test::test_roundtrip_backward65536
    enc::test::test_roundtrip_compressed_repeated
    enc::test::test_roundtrip_monkey
    enc::test::test_roundtrip_quickfox
    enc::test::test_roundtrip_quickfox_repeated
    enc::test::test_roundtrip_ukkonooa
    enc::test::test_roundtrip_x

test result: FAILED$<2>. 18 passed; 11 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
danielrh commented 1 month ago

Ok I think I got to the bottom of it--it was running out of memory due to running too many tests that use lots of memory in parallel. I took some locks if the pointers are sized 32 and it seems to pass every time now.

musicinmybrain commented 1 month ago

It’s surprising that so much memory would be needed, but 0462558646c0d0c346f7e152377b33e9d9d507af does seem to work – thanks!

musicinmybrain commented 1 month ago

It’s surprising that so much memory would be needed, but 0462558 does seem to work – thanks!

I spoke too soon. A build in a local 32-bit chroot was successful, but I still see test failures on Fedora’s i686 builders:

failures:
---- enc::test::test_roundtrip_10x10y stdout ----
thread 'enc::test::test_roundtrip_10x10y' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- enc::test::test_roundtrip_quickfox stdout ----
thread 'enc::test::test_roundtrip_quickfox' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
---- enc::test::test_roundtrip_first_58_bytes_alice stdout ----
thread 'enc::test::test_roundtrip_first_58_bytes_alice' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
---- enc::test::test_roundtrip_quickfox_repeated stdout ----
thread 'enc::test::test_roundtrip_quickfox_repeated' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
---- enc::test::test_roundtrip_ukkonooa stdout ----
thread 'enc::test::test_roundtrip_ukkonooa' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
---- enc::test::test_roundtrip_monkey stdout ----
thread 'enc::test::test_roundtrip_monkey' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
---- enc::test::test_roundtrip_aaabaaaa stdout ----
thread 'enc::test::test_roundtrip_aaabaaaa' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
---- enc::test::test_roundtrip_backward65536 stdout ----
thread 'enc::test::test_roundtrip_backward65536' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
---- enc::test::test_roundtrip_compressed_repeated stdout ----
thread 'enc::test::test_roundtrip_compressed_repeated' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
---- enc::test::test_roundtrip_asyoulik9_5 stdout ----
thread 'enc::test::test_roundtrip_asyoulik9_5' panicked at /usr/share/cargo/registry/alloc-no-stdlib-2.0.4/src/stack_allocator.rs:52:13:
OOM
failures:
    enc::test::test_roundtrip_10x10y
    enc::test::test_roundtrip_aaabaaaa
    enc::test::test_roundtrip_asyoulik9_5
    enc::test::test_roundtrip_backward65536
    enc::test::test_roundtrip_compressed_repeated
    enc::test::test_roundtrip_first_58_bytes_alice
    enc::test::test_roundtrip_monkey
    enc::test::test_roundtrip_quickfox
    enc::test::test_roundtrip_quickfox_repeated
    enc::test::test_roundtrip_ukkonooa
test result: FAILED. 21 passed; 10 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.09s