Open mjc opened 5 years ago
Thank you for reporting this issue. Could you share the code for reproducing this problem in my environment?
Address Sanitizer and/or Thread Sanitizer could be used to pinpoint the memory safety issue, see https://github.com/japaric/rust-san
I'm not sure if they work on Windows, though.
Sure, this is the (rather admittedly terrible as I am quite new to rust) project I'm starting https://github.com/mjc/mame_coalesce. I can try and isolate it a bit more if you want.
I can try and reproduce it on Linux also if need be.
Trying ASan, MemorySanitizer, and ThreadSan now.
EDIT: none of the sanitizers detected anything, sadly.
@mjc Thank you for your information! I will try to reproduce it on my local machine.
zip
crate has recently switched to miniz_oxide as the compression backend. @mjc has anything changed for your program? Is the issue still reproducible?
zip
crate has recently switched to miniz_oxide as the compression backend. @mjc has anything changed for your program? Is the issue still reproducible?
excellent I'll retry today. also finally have a fast linux box to try this on too
On both Linux and windows I don't experience this bug with the new zip crate update that doesn't use this library.
I'm using
zip-rs
with rayon to zip many files concurrently (I think 16x? Whatever rayon ends up using on this machine) This works fine with compression level store, and with bzip2. (Bzip2 is also way, way faster, 150+MB/s instead of 10MB/s, but that is.... another issue.)All of the actual zipping is happening inside each loop iteration that is parallelized: open the source and destination files, io::copy, close the file when done.
When not using
par_iter
it completes successfully (but takes 14 hours to do so on a small test dataset of 283GB.)With deflate (which uses this library,) I run into the below stack trace eventually, on win64.
It takes about an hour or so to run into the issue and doesn't seem to be on the same file every time, so reducing this test case seems difficult.
I've also gotten an illegal instruction once or twice which makes me think there is undefined behavior somewhere in this lib.