solardiz
commented
3 years ago Alternative implementation could store hashes/fingerprints instead of actual candidate passwords. As I read, hashcat brain stores 64-bit hashes. We could, too, for perhaps a sufficiently low rate of false positive matches (and thus of potential false negatives in terms of password cracking, where we'd wrongly skip a non-dupe). This eliminates the need to bother with variable-length entries, and (more importantly) likely lets us store more entries per buffer size. We need to compute some fast hashes anyway, for fast lookup, although we'd need to compute larger hashes to have enough bits for index and for storage.
For short passwords (smaller than our chosen hash storage size), we could store them directly, and allocate and set a bit to indicate that. Then our hashes would be one bit smaller, but we'd have assurance of absolutely no false matches/skips until a certain password length (perhaps 8). This could also speed up processing of such passwords (only need to compute a shorter hash, for the index only).
magnumripper
We currently have full duplicate candidate suppression in the separate
uniquetool, and we also reuse code from there to suppress duplicate input wordlist entries and wordlist rules. However, when we're running through wordlist+rules (maybe after suppression of duplicates within the wordlist and within the rules separately), we only suppress consecutive duplicates (compare against the immediately preceding candidate password).I think we could (optionally) do better (for slow hashes or/and many salts) by maintaining and comparing against a configurable-size buffer of previous candidates.
In terms of implementation, I think this could be a ring buffer and a hash table. In
unique, we use a non-ring buffer and a hash table. We reset them each time the buffer fills up. I think for this new feature we would instead need to be progressively removing old entries from the hash table as the ring buffer's contents are about to be overwritten with new entries. Further, with variable-size entries (likeuniquealready uses for different string lengths), we need to support a case when adding just one new entry requires removal of several previous entries from the hash table first.We can possibly improve the effectiveness of such dupe suppression (percentage suppressed per buffer size) by maintaining (recent) hit counts on entries, so that frequently or/and recently hit entries would be retained for longer even if they're near the overwrite pointer (were entered into the ring buffer long ago). We can either keep them intact (jump over them in our overwrites) or re-add them right away (if needed to avoid gaps that would result from changes in nearby string lengths).
We can also reuse this logic to implement a hashcat brain alike later. Per my reading of hashcat brain documentation, it cannot maintain a maximum size and thus could eventually run out of storage resources and fail. The alternative I suggest here can instead maintain a maximum size, at the expense of imperfect dupe suppression. So it could make sense as an alternative brain implementation, including for use with hashcat if we implement the same protocol.
In fact, my motivation behind suggesting this feature now is needing to run some slow and lengthy wordlist+rules attacks where the resulting candidate password streams are too large to comfortably pass them through
unique(orrlingorduplicut) in advance and store the results, or to use hashcat brain. Say, terabytes. I am thinking that with a ring buffer size larger than input wordlist size (say, a few GB or tens of GB, so still fitting in RAM) I could probably suppress most duplicates without incurring as much storage and processing overhead, and also with the processing overhead comfortably spread over the attack duration. I'd first simulate this on a similar-looking but much smaller wordlist with the same ruleset, comparing the percentage of dupes suppressed byuniqueor the like vs. this opportunistic approach with a ring buffer size scaled to the smaller wordlist's size (e.g., 10x the wordlist size if that's what I'd be able to afford in the target use case). Once confirmed that only acceptably few dupes are left with the opportunistic approach in this scaled-down simulation, I'd proceed to run the real attack.