search

image-rs / deflate-rs

An implementation of a DEFLATE encoder in rust
Apache License 2.0
53 stars 14 forks source link

Internal assertion fails at images larger than 4800×4800 #44

Closed ElectricCoffee closed 4 years ago

ElectricCoffee commented 4 years ago

I recently made use of this library in an effort to make a mandelbrot set, but in my quest to achieve ever higher resolutions I ran into a problem: an internal assertion fails if I exceed 4800×4800 pixels.

You can find the relevant code here: https://github.com/ElectricCoffee/mandelbrot

The error that is generated goes like this:

PS D:\Code\rust\mandelbrot> cargo run
   Compiling num-complex v0.2.4
   Compiling mandelbrot v0.1.0 (D:\Code\rust\mandelbrot)
    Finished dev [unoptimized + debuginfo] target(s) in 4.04s
     Running `target\debug\mandelbrot.exe`
Generating mandelbrot_8000x8000.png.
Generating image data, hold tight...
Flattening...
Writing image data...
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `192008000`,
 right: `192008189`', <::std::macros::panic macros>:5:6
stack backtrace:
   0: backtrace::backtrace::trace_unsynchronized
             at C:\Users\VssAdministrator\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.40\src\backtrace\mod.rs:66
   1: std::sys_common::backtrace::_print_fmt
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\sys_common\backtrace.rs:77
   2: std::sys_common::backtrace::_print::{{impl}}::fmt
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\sys_common\backtrace.rs:59
   3: core::fmt::write
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libcore\fmt\mod.rs:1052
   4: std::io::Write::write_fmt<std::sys::windows::stdio::Stderr>
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\io\mod.rs:1426
   5: std::sys_common::backtrace::_print
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\sys_common\backtrace.rs:62
   6: std::sys_common::backtrace::print
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\sys_common\backtrace.rs:49
   7: std::panicking::default_hook::{{closure}}
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:204
   8: std::panicking::default_hook
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:224
   9: std::panicking::rust_panic_with_hook
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:472
  10: std::panicking::begin_panic_handler
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:380
  11: std::panicking::begin_panic_fmt
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:334
  12: deflate::writer::compress_until_done<alloc::vec::Vec<u8>>
             at <::std::macros::panic macros>:5
  13: deflate::writer::ZlibEncoder<alloc::vec::Vec<u8>>::output_all<alloc::vec::Vec<u8>>
             at C:\Users\Electric Coffee\.cargo\registry\src\github.com-1ecc6299db9ec823\deflate-0.8.3\src\writer.rs:205
  14: deflate::writer::ZlibEncoder<alloc::vec::Vec<u8>>::finish<alloc::vec::Vec<u8>>
             at C:\Users\Electric Coffee\.cargo\registry\src\github.com-1ecc6299db9ec823\deflate-0.8.3\src\writer.rs:212
  15: png::encoder::Writer<std::io::buffered::BufWriter<std::fs::File>>::write_image_data<std::io::buffered::BufWriter<std::fs::File>>
             at C:\Users\Electric Coffee\.cargo\registry\src\github.com-1ecc6299db9ec823\png-0.16.1\src\encoder.rs:172
  16: mandelbrot::main
             at .\src\main.rs:73
  17: std::rt::lang_start::{{closure}}<()>
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\src\libstd\rt.rs:67
  18: std::rt::lang_start_internal::{{closure}}
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\rt.rs:52
  19: std::panicking::try::do_call<closure-0,i32>
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:305
  20: panic_unwind::__rust_maybe_catch_panic
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libpanic_unwind\lib.rs:86
  21: std::panicking::try
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:281
  22: std::panic::catch_unwind
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panic.rs:394
  23: std::rt::lang_start_internal
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\rt.rs:51
  24: std::rt::lang_start<()>
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\src\libstd\rt.rs:67
  25: main
  26: invoke_main
             at d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  27: __scrt_common_main_seh
             at d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
  28: BaseThreadInitThunk
  29: RtlUserThreadStart
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: process didn't exit successfully: `target\debug\mandelbrot.exe` (exit code: 101)
PS D:\Code\rust\mandelbrot>

The above error is for an image 8000×8000 px in size

HeroicKatora commented 4 years ago

I can't reproduce this, and the error happens to be in deflate. Can you provide the contents of Cargo.lock or ensure you are running on the lastest versions?

nabijaczleweli commented 4 years ago

Repro on buster w/rustc 1.41.1 (f3e1a954d 2020-02-24):

nabijaczleweli@tarta:~/uwu$ git clone https://github.com/ElectricCoffee/mandelbrot
Cloning into 'mandelbrot'...
remote: Enumerating objects: 55, done.
remote: Counting objects: 100% (55/55), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 55 (delta 24), reused 35 (delta 5), pack-reused 0
Unpacking objects: 100% (55/55), done.
nabijaczleweli@tarta:~/uwu$ cd mandelbrot/
nabijaczleweli@tarta:~/uwu/mandelbrot$ cargo build
  Downloaded png v0.16.1
  Downloaded deflate v0.8.3
   Compiling autocfg v1.0.0
   Compiling byteorder v1.3.4
   Compiling adler32 v1.0.4
   Compiling bitflags v1.2.1
   Compiling crc32fast v1.2.0
   Compiling cfg-if v0.1.10
   Compiling inflate v0.4.5
   Compiling num-traits v0.2.11
   Compiling num-complex v0.2.4
   Compiling deflate v0.8.3
   Compiling png v0.16.1
   Compiling mandelbrot v0.1.0 (/home/nabijaczleweli/uwu/mandelbrot)
    Finished dev [unoptimized + debuginfo] target(s) in 6.57s
nabijaczleweli@tarta:~/uwu/mandelbrot$ ./target/debug/mandelbrot
Generating mandelbrot_8000x8000.png.
Generating image data, hold tight...
Flattening...
Writing image data...
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `192008000`,
 right: `192008189`', /home/nabijaczleweli/.cargo/registry/src/github.com-1ecc6299db9ec823/deflate-0.8.3/src/writer.rs:54:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
nabijaczleweli@tarta:~/uwu/mandelbrot$ export RUST_BACKTRACE=full
nabijaczleweli@tarta:~/uwu/mandelbrot$ ./target/debug/mandelbrot
Generating mandelbrot_8000x8000.png.
Generating image data, hold tight...
Flattening...
Writing image data...
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `192008000`,
 right: `192008189`', /home/nabijaczleweli/.cargo/registry/src/github.com-1ecc6299db9ec823/deflate-0.8.3/src/writer.rs:54:5
stack backtrace:
   0:     0x5603eb2f21c4 - backtrace::backtrace::libunwind::trace::he25250f78ba1020d
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x5603eb2f21c4 - backtrace::backtrace::trace_unsynchronized::hd7e25f35da233a1e
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x5603eb2f21c4 - std::sys_common::backtrace::_print_fmt::h5578a566a4c2ae4e
                               at src/libstd/sys_common/backtrace.rs:84
   3:     0x5603eb2f21c4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd4f87054ec86e936
                               at src/libstd/sys_common/backtrace.rs:61
   4:     0x5603eb30b0dc - core::fmt::write::h393d55821fa8e2cb
                               at src/libcore/fmt/mod.rs:1025
   5:     0x5603eb2f0807 - std::io::Write::write_fmt::hc6645302c23e4504
                               at src/libstd/io/mod.rs:1426
   6:     0x5603eb2f41ae - std::sys_common::backtrace::_print::he8a0a3ebec70e1d6
                               at src/libstd/sys_common/backtrace.rs:65
   7:     0x5603eb2f41ae - std::sys_common::backtrace::print::hc2635fd6c7743ade
                               at src/libstd/sys_common/backtrace.rs:50
   8:     0x5603eb2f41ae - std::panicking::default_hook::{{closure}}::h9849388798b72b2b
                               at src/libstd/panicking.rs:193
   9:     0x5603eb2f3ea1 - std::panicking::default_hook::h66e3afff11b02e47
                               at src/libstd/panicking.rs:210
  10:     0x5603eb2f482b - std::panicking::rust_panic_with_hook::hf6d05969fcffff03
                               at src/libstd/panicking.rs:471
  11:     0x5603eb2f43de - rust_begin_unwind
                               at src/libstd/panicking.rs:375
  12:     0x5603eb2f4320 - std::panicking::begin_panic_fmt::h0ba4167e9ba9b09d
                               at src/libstd/panicking.rs:326
  13:     0x5603eb29b8d3 - deflate::writer::compress_until_done::h7129f876d1e6c9dc
                               at /home/nabijaczleweli/uwu/mandelbrot/<::std::macros::panic macros>:9
  14:     0x5603eb29baeb - deflate::writer::ZlibEncoder<W>::output_all::hec8d1c8f7a455a51
                               at /home/nabijaczleweli/.cargo/registry/src/github.com-1ecc6299db9ec823/deflate-0.8.3/src/writer.rs:205
  15:     0x5603eb29bfe6 - deflate::writer::ZlibEncoder<W>::finish::hcdfaf89500c35362
                               at /home/nabijaczleweli/.cargo/registry/src/github.com-1ecc6299db9ec823/deflate-0.8.3/src/writer.rs:212
  16:     0x5603eb29cfc6 - png::encoder::Writer<W>::write_image_data::hdb84dad948ef5a21
                               at /home/nabijaczleweli/.cargo/registry/src/github.com-1ecc6299db9ec823/png-0.16.1/src/encoder.rs:172
  17:     0x5603eb2a5c55 - mandelbrot::main::he71202c5d3a2e124
                               at src/main.rs:73
  18:     0x5603eb2a0a70 - std::rt::lang_start::{{closure}}::hde55f19841f12863
                               at /rustc/f3e1a954d2ead4e2fc197c7da7d71e6c61bad196/src/libstd/rt.rs:67
  19:     0x5603eb2f4273 - std::rt::lang_start_internal::{{closure}}::h4aedff5d9863b36b
                               at src/libstd/rt.rs:52
  20:     0x5603eb2f4273 - std::panicking::try::do_call::h0630fe3da32a089b
                               at src/libstd/panicking.rs:292
  21:     0x5603eb2f63ca - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:78
  22:     0x5603eb2f4cc0 - std::panicking::try::hdfbe2fd873ac646c
                               at src/libstd/panicking.rs:270
  23:     0x5603eb2f4cc0 - std::panic::catch_unwind::hd7a9c10fa2bee1bc
                               at src/libstd/panic.rs:394
  24:     0x5603eb2f4cc0 - std::rt::lang_start_internal::hb8081e0e7e42ac0d
                               at src/libstd/rt.rs:51
  25:     0x5603eb2a0a49 - std::rt::lang_start::hb072ac6bcc199098
                               at /rustc/f3e1a954d2ead4e2fc197c7da7d71e6c61bad196/src/libstd/rt.rs:67
  26:     0x5603eb2a5d3a - main
  27:     0x7f4f357d109b - __libc_start_main
  28:     0x5603eb29916a - _start
  29:                0x0 - <unknown>
Cargo.lock 
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
[[package]]
name = "adler32"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d2e7343e7fc9de883d1b0341e0b13970f764c14101234857d2ddafa1cb1cac2"

[[package]]
name = "autocfg"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d"

[[package]]
name = "bitflags"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"

[[package]]
name = "byteorder"
version = "1.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"

[[package]]
name = "cfg-if"
version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"

[[package]]
name = "crc32fast"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba125de2af0df55319f41944744ad91c71113bf74a4646efff39afe1f6842db1"
dependencies = [
 "cfg-if",
]

[[package]]
name = "deflate"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "050ef6de42a33903b30a7497b76b40d3d58691d4d3eec355348c122444a388f0"
dependencies = [
 "adler32",
 "byteorder",
]

[[package]]
name = "inflate"
version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1cdb29978cc5797bd8dcc8e5bf7de604891df2a8dc576973d71a281e916db2ff"
dependencies = [
 "adler32",
]

[[package]]
name = "mandelbrot"
version = "0.1.0"
dependencies = [
 "num-complex",
 "png",
]

[[package]]
name = "num-complex"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6b19411a9719e753aff12e5187b74d60d3dc449ec3f4dc21e3989c3f554bc95"
dependencies = [
 "autocfg",
 "num-traits",
]

[[package]]
name = "num-traits"
version = "0.2.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c62be47e61d1842b9170f0fdeec8eba98e60e90e5446449a0545e5152acd7096"
dependencies = [
 "autocfg",
]

[[package]]
name = "png"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "46060468187c21c00ffa2a920690b29997d7fd543f5a4d400461e4a7d4fccde8"
dependencies = [
 "bitflags",
 "crc32fast",
 "deflate",
 "inflate",
]
HeroicKatora commented 4 years ago

Ah, it reproduces on debug builds only which I hadn't tried due to expecting computation heavy code. That difference is slightly concerning as that is a debug_assert, see here.

HeroicKatora commented 4 years ago

I'm moving the issue though as this seems to be an internal issue of deflate.

oyvindln commented 4 years ago

If it compresses and decompresses fine in release it may just an error in the debug check rather than anything wrong with the compression, otherwise errors usually leads to corrupted files. It's a leftover from development to check validate the number of input bytes processed, so maybe the large size causes that to be off, I will look into it.

ElectricCoffee commented 4 years ago

Oddly enough, running the code in release mode causes no issues at all. The code even spits out perfectly good images without any corruption.

@HeroicKatora the Cargo.lock should be in the linked repo