Open kmglennan opened 7 months ago
Well that's fun... 🥹 I suspect this comes from one of our used crates unless this is a bug from rust itself since afaik there is no funny pointer magic nor unsafe code in this project.
I suspect this is from either crossbeam or pdf crate since those are the more complex ones used in this project that might use pointers internally. Very likely crossbeam, having 128 workers would give it plenty of opportunities for weird racy behavior which i suspect caused this.
Pdf crate is sorta exempt as the only operations we use from it with data shared between threads are non-mutable. I'll give the crate a read to see if there is anything suspect as well as try to reproduce this somehow.
For now it might be useful to know how many cores/threads your machine has.
The machine this was running on has 24 cores. Specs are as follows:
Model Name: Mac Studio
Model Identifier: Mac14,14
Model Number: Z180000FZX/A
Chip: Apple M2 Ultra
Total Number of Cores: 24 (16 performance and 8 efficiency)
Memory: 192 GB
System Firmware Version: 10151.61.4
OS Loader Version: 10151.61.4
So i've set my server to run this command for now together with some more debug information and I'll come back to it in a day or two and hope to see a similar crash.
I strongly suspect this is crossbeam at this point since it has historically seen similar issues such as CVE-2021-32810 and CVE-2022-23639. I do believe we are right above the patch version for the former however since it was supposedly fixed in 0.8.1, and pdfrip is on 0.8.2.
We are affected by the latter, which might be the cause of this although double free is not mentioned among the consequences (but it might just be included in "Data race").
A solution for this is probably to either A: Update the crossbeam dependency or B. Use channels from some other library that are less racy.
@Pommaq, have you tried https://github.com/zesterer/flume? Appears to be faster than crossbeam
channels. We can probably try switching for better performance and probably fix this bug. I say "probably" since an implementation with better performance would probably contain funny magic code increasing chances of race conditions and the like.
Never heard of it, but a quick 10 second look indicates it's similar enough to basically just be a drop-in replacement, I do like how it doesn't use unsafe
, so this is probably better.