Crash resulting from malloc error

[kmglennan]

$ pdfrip -f pw.pdf -n 128 default-query --min-length 7 --max-length 7
           .___ _____       .__        
______   __| _// ____\______|__|_____  
\____ \ / __ |\   __\\_  __ \  \____ \ 
|  |_> > /_/ | |  |   |  | \/  |  |_> >
|   __/\____ | |__|   |__|  |__|   __/ 
|__|        \/                 |__|    2.0.1

 2024-01-17T01:29:12.322Z INFO  pdfrip::core::engine > Starting password cracking job...
⠚ [2d 15:01:46] [███████████████░░░░░░░░░░░░░░░░░░░░░░░░░] 30932726000/78364164096 39% 136332/s ETA: 4d
pdfrip(21967,0x16c9a7000) malloc: *** error for object 0x600001910f50: pointer being freed was not allocated
pdfrip(21967,0x16c9a7000) malloc: *** set a breakpoint in malloc_error_break to debug
pdfrip(21967,0x1725c3000) malloc: Heap corruption detected, free list is damaged at 0x60000190c040
*** Incorrect guard value: 10107014426694143842
pdfrip(21967,0x1725c3000) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6

[Pommaq]

Well that's fun... 🥹 I suspect this comes from one of our used crates unless this is a bug from rust itself since afaik there is no funny pointer magic nor unsafe code in this project.

I suspect this is from either crossbeam or pdf crate since those are the more complex ones used in this project that might use pointers internally. Very likely crossbeam, having 128 workers would give it plenty of opportunities for weird racy behavior which i suspect caused this.

Pdf crate is sorta exempt as the only operations we use from it with data shared between threads are non-mutable. I'll give the crate a read to see if there is anything suspect as well as try to reproduce this somehow.

For now it might be useful to know how many cores/threads your machine has.

[kmglennan]

The machine this was running on has 24 cores. Specs are as follows:

  Model Name: Mac Studio
  Model Identifier: Mac14,14
  Model Number: Z180000FZX/A
  Chip: Apple M2 Ultra
  Total Number of Cores: 24 (16 performance and 8 efficiency)
  Memory: 192 GB
  System Firmware Version: 10151.61.4
  OS Loader Version: 10151.61.4

[Pommaq]

So i've set my server to run this command for now together with some more debug information and I'll come back to it in a day or two and hope to see a similar crash.

I strongly suspect this is crossbeam at this point since it has historically seen similar issues such as CVE-2021-32810 and CVE-2022-23639. I do believe we are right above the patch version for the former however since it was supposedly fixed in 0.8.1, and pdfrip is on 0.8.2.

We are affected by the latter, which might be the cause of this although double free is not mentioned among the consequences (but it might just be included in "Data race").

A solution for this is probably to either A: Update the crossbeam dependency or B. Use channels from some other library that are less racy.

[mufeedvh]

@Pommaq, have you tried https://github.com/zesterer/flume? Appears to be faster than crossbeam channels. We can probably try switching for better performance and probably fix this bug. I say "probably" since an implementation with better performance would probably contain funny magic code increasing chances of race conditions and the like.

[Pommaq]

Never heard of it, but a quick 10 second look indicates it's similar enough to basically just be a drop-in replacement, I do like how it doesn't use unsafe, so this is probably better.