option: sort a wordlist by mask, then by natural sort order

[roycewilliams]

It would be very useful to be able to sort a wordlist first by length, then by mask, then by natural sort order.

Example:

2345
z5678
1234
C5309
9999a
2345a
1234a
!6666
z9933
a1234

... would sort as:

1234
2345
1234a
2345a
9999a
a1234
z5678
z9933
C5309
!6666

Suggested mask order: ?d?l?u?s, to match the rough order of frequency groups.

It would be useful for this option to support sorting both raw wordlists and potfiles. Automatic interpretation of $HEX would also be good.

[hops]

I have no intention to support sorting of potfiles. The p in pack is for password, not for potfile. $HEX support has been added with 876d7cd

[roycewilliams]

Hey, our first disagreement! :)

I would like to advocate for potfile awareness in PACK tools - both in this potential new tool, and generally.

Yes, the 'p' stands for 'password' ... but the 'c' stands for 'cracking'. :wink: The vast majority of PACK users are spending a lot of time managing the flow of data and results among wordlists, masks, rules, and potfiles. The whole purpose of PACK is to make those workflows easier. Being able to work with potfiles directly would be a significant timesaver for many workflows.

Before PACK, people had to build their own tools to sort masks. PACK became the standard because it solved a common use case, and solved it well. In my opinion, if there is something that many password crackers commonly do to analyze or manage their work, it should be considered to be in scope for PACK.

Just as an example, there isn't much in PACK today that works with rules. But if there was a good way to solve the problem of extracting rules from hashcat rules-debug files, and PACK implemented it, then it would save many users the time and effort of "reinventing the wheel". There's no 'r' in 'pack', but rules are clearly also part of the password-cracking workflow, and such a tool would be a big help to the password-cracking community.

I believe that pack2 should be more ambitious than just "like for like", and consider new ways to innovate in the password-cracking space.

[hops]

I agree that pack2 shouldn't be a 1:1 rewrite and then call it done. As you know there are already plans that go beyond what PACK is capable of. I should have explained why I have no intention to support potfiles. They are really hard to parse correctly! Sure I can simply find the right most : and everything after it's position is the password. That's probably gonna work in most cases. But what if : wasn't the separator? Or even better if it was at one point but a later cracking session writing to the same file used \t instead? Or --outfile-autohex-disable was active? There's also nothing that would tell what type of hash it is. And even then it would mean to implement a parser for every type supported in hashcat.

I might consider going with the easiest solution (finding the last : of a line) and call it best effort with a big red warning.

[Waffle2]

This exact issue is why mdxfind uses the hash type in output, and why mdsplit exists. Potfiles are, at best, unreliable. At worst, they are deliberately misleading. I have had hundreds of hash "solutions" which could not be duplicated (yes, it's true most of them are due to $HEX[] issues, and nulls in strings). Please do not attempt to encourage their use. Solve hashes and indicate the hash type - which at least gives you a hope of both parsing the password and salt from the hash, and (more important) validating the solution in a future pass, long after you have forgotten what type of hash it is. I guess I should re-title this to "potfiles considered harmful" :-)

[roycewilliams]

I might consider going with the easiest solution (finding the last : of a line) and call it best effort with a big red warning.

That seems totally reasonable - I'm just looking to support the simplest, common case. The most common use case - NTLM cracking - has a very simple resulting potfile that can be easily parsed. And if someone is overriding the : separator, they will be savvy enough to heed this warning (and to manage their own local setup accordingly).

Please do not attempt to encourage their use.

I appreciate the purity of mission, but the likelihood that potfiles are going to go away any time soon is about zero. And even if people do not generate potfiles themselves, they're very likely to have to deal with them from other sources. We can either leave each such user to fend for themselves, reinventing the "clean up a crappy potfile" wheel poorly, or we can try to centralize our wisdom about how to deal with them in a way that's a force multiplier, freeing up more people to focus more time on the actual work of password cracking, instead of futzing with their sub-optimal, locally-created tools and swearing. :)

Put another way: we do not encourage the use of bad hashing methods - but we darned well have good tools to work with them coughMDXfindcough. ;)

pack2's goals are, of course, up to you.

[Waffle2]

Maybe it really is time, as with $HEX[], to write a full "potfiles considered harmful", and ask for the change? I took a lot of heat back in the day for $HEX. I don't mind being called names again :-)

[roycewilliams]

Indeed. I support the idea - and I want to be clear that I think that we do collectively need to grow beyond potfiles. I'm just jonesin' for some solid toolsmithing in the meantime. :)