search

Frommi / miniz_oxide

Rust replacement for miniz
MIT License
168 stars 48 forks source link

'attempt to add with overflow' during deflate. #74

Closed itzurabhi closed 4 years ago

itzurabhi commented 4 years ago

Found as part of the Rust compiler build. miniz version - miniz_oxide-0.3.5 Rust version tag : release v1.41.1 Channel : nightly config.toml

[llvm]
ninja = true
link-jobs = 1
version-suffix = "-rust"
release-debuginfo = true
use-libcxx = true
use-linker = "lld"

[rust]
optimize = true
debug = true
debuginfo-level = 2
debuginfo-level-rustc = 2
debuginfo-level-std = 2
debuginfo-level-tools = 2
channel = "nightly"

[build]
extended = true
verbose = 1

during x.py install

Dist rustc stage2 (x86_64-unknown-linux-gnu)
running: "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=Rust-is-ready-to-roll." "--image-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-image" "--work-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist" "--output-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/dist" "--non-installed-overlay" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-overlay" "--package-name=rustc-nightly-x86_64-unknown-linux-gnu" "--component-name=rustc" "--legacy-manifest-dirs=rustlib,cargo"
thread '<unnamed>' panicked at 'attempt to add with overflow', /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:1669:31
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1028
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1412
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:188
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:205
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:464
  11: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:373
  12: rust_begin_unwind
             at src/libstd/panicking.rs:302
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:139
  14: core::panicking::panic
             at src/libcore/panicking.rs:70
  15: miniz_oxide::deflate::core::compress_inner
           at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:0
  16: miniz_oxide::deflate::core::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:2070
  17: miniz_oxide::deflate::stream::deflate
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/stream.rs:52
  18: <flate2::ffi::rust::Deflate as flate2::ffi::DeflateBackend>::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/ffi/rust.rs:149
  19: flate2::mem::Compress::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/mem.rs:297
  20: flate2::mem::Compress::compress_vec
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/mem.rs:322
  21: <flate2::mem::Compress as flate2::zio::Ops>::run_vec
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:54
  22: flate2::zio::Writer<W,D>::write_with_status
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:217
  23: <flate2::zio::Writer<W,D> as std::io::Write>::write
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:256
  24: <flate2::gz::write::GzEncoder<W> as std::io::Write>::write
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/gz/write.rs:149
  25: std::io::Write::write_all
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/io/mod.rs:1341
  26: <installer::tarballer::RayonTee<A,B> as std::io::Write>::write_all::{{closure}}
             at src/tools/rust-installer/src/tarballer.rs:175
  27: rayon_core::join::join::call::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/join/mod.rs:103
  28: rayon_core::join::join_context::call_b::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/join/mod.rs:130
  29: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::call::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:113
  30: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:317
  31: std::panicking::try::do_call
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:287
  32: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  33: std::panicking::try
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:265
  34: std::panic::catch_unwind
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:396
  35: rayon_core::unwind::halt_unwinding
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/unwind.rs:17
  36: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:119
  37: rayon_core::job::JobRef::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:59
  38: rayon_core::registry::WorkerThread::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:735
  39: rayon_core::registry::WorkerThread::wait_until_cold
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:719
  40: rayon_core::registry::WorkerThread::wait_until
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:693
  41: rayon_core::registry::main_loop
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:813
  42: rayon_core::registry::ThreadBuilder::run
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:58
  43: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:103

command did not execute successfully: "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=Rust-is-ready-to-roll." "--image-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-image" "--work-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist" "--output-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/dist" "--non-installed-overlay" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-overlay" "--package-name=rustc-nightly-x86_64-unknown-linux-gnu" "--component-name=rustc" "--legacy-manifest-dirs=rustlib,cargo"
expected success, got: signal: 4

https://github.com/solana-labs/solana/issues/5411 looks similar.

oyvindln commented 4 years ago

Yeah seems similar. The backtrace points to line 0 in core.rs for some reason, so I will try to reproduce to find which variable overflows.

oyvindln commented 4 years ago

Sorry for taking a while to try to solve this. Are you able to test if it works on the latest commit?

I've changed the buffers to use usize for indexing, and it fixed the overflow when testing with a huge file for me, so hopefully it solves this too.

itzurabhi commented 4 years ago

Thanks for fixing it.

I didn't test the library standalone,I got the error while building rustc itself (package step) .

I'll try and update here.

dylanmckay commented 4 years ago

I'm also hitting this whilst compiling rustc - is it possible to publish the 0.3.7 release to crates.io?

I'd like to submit a PR to flate2, then subsequently to rustc, fixing the overflow errors during rustc x.py install.

oyvindln commented 4 years ago

Ah, thought I had put it up, seems I put up the c api part instead.