pmarks
commented
5 months ago I was able to reproduce this on Linux with Rust 1.73, and a similarly sized file. Here's the backtrace:
#1 0x00005630f3b86148 in <alloc::vec::Vec<u8,A> as std::io::copy::BufferedWriterSpec>::copy_from ()
#2 0x00005630f3b8712f in lz4::test::speed_bug ()
#3 0x00005630f3b802a9 in core::ops::function::FnOnce::call_once ()
#4 0x00005630f3bc3d5f in core::ops::function::FnOnce::call_once () at library/core/src/ops/function.rs:250
#5 test::__rust_begin_short_backtrace () at library/test/src/lib.rs:626
...```
It appears to be back to norma speed in Rust 1.75, so I guess it was fixed? I can't find an obvious rust issue that matches this phenotype, so 🤷 . Closing - please re-open if you have any more insights.
bonsairobo
I'm doing some benchmarks with a large (1.7GB) CSV file, and it seems that upgrading from rust 1.71 to 1.73 has caused a significant performance regression for
lz4::Decoder.Some results:
Rust 1.72
Rust 1.73
^ I killed the benchmark during decompression because it took over 11 minutes.
benchmark code
```rust for level in [1, 4, 7] { println!("lz4 {}...", level); let mut in_f = std::fs::File::open(in_f_path).unwrap(); let start_size = in_f.metadata().unwrap().len(); let now = Instant::now(); let out_path = format!("out-{}.lz4", level); let out_f = std::fs::File::create(&out_path).unwrap(); let mut encoder = EncoderBuilder::new().level(level).build(out_f).unwrap(); std::io::copy(&mut in_f, &mut encoder).unwrap(); let (output, result) = encoder.finish(); if let Err(e) = result { println!("{:#?}", e); } let elapsed = now.elapsed().as_millis(); let end_size = output.metadata().unwrap().len(); println!( "[lz4 {}] Compression: {:.4}%, Time: {:.2}s", level, 100.0 * end_size as f32 / start_size as f32, elapsed as f32 / 1000.0, ); let file_bytes = std::fs::read(&out_path).unwrap(); let decode_start = Instant::now(); let mut decoder = lz4::Decoder::new(Cursor::new(file_bytes)).unwrap(); let mut uncompressed_buf: Vec