Validate that duplicate salt removal is working for all formats

[jfoug]

We should look for salted formats that do not remove dupe salts properly.

To do this, we can take a set of X unique salts. Dupe the file 5 times, run it and make sure that jtr ONLY sees X salts and not 5X, or any value > X. I know frank recently did some testing on the --list=format-tests Was this behavior 'looked' for when testing. In other words, if there were 4 unique salts, but 16 lines in the input file, are we sure that jtr only displays 4 salts?

I do not have time to do all of this myself, it was just something I thought about, when working on the new zip2 format. This can happen if a format designs a salt in a manner that an exact bit by bit salt value is returned for any identical hash. We did catch some cannonical failures, but did we catch all ?

NOPE. I just tested with pkzip (a format I was pretty sure does not do salt dupe removal properly). --list=format=tests returns 10 lines, but there are 9 hashes (one is a dupe). JtR shows 10 salts. I duplicated the file data 4 times, and it lists 40 salts. So pkzip format needs salt_dup fixed. Can someone carry this forward and at least list which formats are NOT properly salt dupe detecting. When you find Update findings in this message (both proper working and failures, so we known when we have covered them all). Also, could the logic be different on different builds (64 bit vs 32, SIMD vs non, etc? I do not have this answer).

Format's dupe salt removal logic:

FAILURES:
pkzip  (FAILS, 32 bit SIMD). 

WORKING    (also list build 'type')

Unknown, Still to be tested:
 (when tested, remove them from this list and place into 1 of the above lists)

7z_fmt_plug.c
agilekeychain_fmt_plug.c
aix_smd5_fmt_plug.c
aix_ssha_fmt_plug.c
androidfde_fmt_plug.c
asaMD5_fmt_plug.c
BFEgg_fmt_plug.c
bitcoin_fmt_plug.c
blackberry_ES10_fmt_plug.c
blockchain_fmt_plug.c
chap_fmt_plug.c
citrix_ns_fmt_plug.c
clipperz_srp_fmt_plug.c
cloudkeychain_fmt_plug.c
crc32_fmt_plug.c
crypt-sha1_fmt_plug.c
cryptsha256_fmt_plug.c
cryptsha512_fmt_plug.c
cuda_cryptmd5_fmt_plug.c
cuda_cryptsha256_fmt_plug.c
cuda_cryptsha512_fmt_plug.c
cuda_mscash_fmt_plug.c
cuda_mscash2_fmt_plug.c
cuda_phpass_fmt_plug.c
cuda_pwsafe_fmt_plug.c
cuda_rawsha512_fmt_plug.c
cuda_wpapsk_fmt_plug.c
cuda_xsha512_fmt_plug.c
django_fmt_plug.c
django_scrypt_fmt_plug.c
DMD5_fmt_plug.c
dmg_fmt_plug.c
DOMINOSEC_fmt_plug.c
dragonfly3_fmt_plug.c
dragonfly4_fmt_plug.c
drupal7_fmt_plug.c
ecryptfs_fmt_plug.c
efs_fmt_plug.c
encfs_fmt_plug.c
EPI_fmt_plug.c
episerver_fmt_plug.c
FGT_fmt_plug.c
formspring_fmt_plug.c
gost_fmt_plug.c
gpg_fmt_plug.c
haval_fmt_plug.c
HDAA_fmt_plug.c
hmacMD5_fmt_plug.c
hmacSHA1_fmt_plug.c
hmacSHA224_fmt_plug.c
hmacSHA256_fmt_plug.c
hmacSHA384_fmt_plug.c
hmacSHA512_fmt_plug.c
hmailserver_fmt_plug.c
ike_fmt_plug.c
IPB2_fmt_plug.c
keepass_fmt_plug.c
keychain_fmt_plug.c
keyring_fmt_plug.c
keystore_fmt_plug.c
known_hosts_fmt_plug.c
KRB4_fmt_plug.c
KRB5_fmt_plug.c
krb5-18_fmt_plug.c
krb5pa-md5_fmt_plug.c
krb5pa-sha1_fmt_plug.c
kwallet_fmt_plug.c
lastpass_fmt_plug.c
lastpass_sniffed_fmt_plug.c
lotus5_fmt_plug.c
lotus85_fmt_plug.c
luks_fmt_plug.c
md2_fmt_plug.c
md4_gen_fmt_plug.c
mediawiki_fmt_plug.c
mongodb_fmt_plug.c
mozilla_fmt_plug.c
mscash1_fmt_plug.c
mscash2_fmt_plug.c
MSCHAPv2_bs_fmt_plug.c
mssql05_fmt_plug.c
mssql12_fmt_plug.c
mssql-old_fmt_plug.c
mysql_fmt_plug.c
mysql_netauth_fmt_plug.c
mysqlSHA1_fmt_plug.c
net_md5_fmt_plug.c
net_sha1_fmt_plug.c
NETLM_fmt_plug.c
NETLMv2_fmt_plug.c
NETNTLM_bs_fmt_plug.c
NETNTLMv2_fmt_plug.c
NETSPLITLM_fmt_plug.c
NS_fmt_plug.c
nsldap_fmt_plug.c
NT_fmt_plug.c
nt2_fmt_plug.c
ntlmv1_mschapv2_fmt_plug.c
nukedclan_fmt_plug.c
o5logon_fmt_plug.c
odf_fmt_plug.c
office_fmt_plug.c
oldoffice_fmt_plug.c
opencl_agilekeychain_fmt_plug.c
opencl_bf_fmt_plug.c
opencl_blockchain_fmt_plug.c
opencl_cryptmd5_fmt_plug.c
opencl_cryptsha256_fmt_plug.c
opencl_cryptsha512_fmt_plug.c
opencl_DES_fmt_plug.c
opencl_dmg_fmt_plug.c
opencl_encfs_fmt_plug.c
opencl_gpg_fmt_plug.c
opencl_keychain_fmt_plug.c
opencl_keyring_fmt_plug.c
opencl_krb5pa-md5_fmt_plug.c
opencl_krb5pa-sha1_fmt_plug.c
opencl_lotus5_fmt_plug.c
opencl_mscash2_fmt_plug.c
opencl_mysqlsha1_fmt_plug.c
opencl_nsldaps_fmt_plug.c
opencl_nt_fmt_plug.c
opencl_ntlmv2_fmt_plug.c
opencl_o5logon_fmt_plug.c
opencl_odf_aes_fmt_plug.c
opencl_odf_fmt_plug.c
opencl_office2007_fmt_plug.c
opencl_office2010_fmt_plug.c
opencl_office2013_fmt_plug.c
opencl_pbkdf2_hmac_sha256_fmt_plug.c
opencl_pbkdf2_hmac_sha512_fmt_plug.c
opencl_phpass_fmt_plug.c
opencl_pwsafe_fmt_plug.c
opencl_rakp_fmt_plug.c
opencl_rar_fmt_plug.c
opencl_rawmd4_fmt_plug.c
opencl_rawmd5_fmt_plug.c
opencl_rawsha1_fmt_plug.c
opencl_rawsha256_fmt_plug.c
opencl_rawsha512_fmt_plug.c
opencl_strip_fmt_plug.c
opencl_sxc_fmt_plug.c
opencl_wpapsk_fmt_plug.c
opencl_zip_fmt_plug.c
openssl_enc_fmt_plug.c
oracle_fmt_plug.c
oracle11_fmt_plug.c
osc_fmt_plug.c
panama_fmt_plug.c
pbkdf2_hmac_sha256_fmt_plug.c
pbkdf2-hmac-sha1_fmt_plug.c
pbkdf2-hmac-sha512_fmt_plug.c
pdf_fmt_plug.c
pfx_fmt_plug.c
phpassMD5_fmt_plug.c
PHPS_fmt_plug.c
pixMD5_fmt_plug.c
PO_fmt_plug.c
postgres_fmt_plug.c
pst_fmt_plug.c
putty_fmt_plug.c
pwsafe_fmt_plug.c
racf_fmt_plug.c
radmin_fmt_plug.c
rakp_fmt_plug.c
rar_fmt_plug.c
rar5_fmt_plug.c
rawBLAKE2_512_fmt_plug.c
rawKeccak_256_fmt_plug.c
rawKeccak_512_fmt_plug.c
rawMD4_fmt_plug.c
rawMD5_fmt_plug.c
rawmd5u_fmt_plug.c
rawSHA0_fmt_plug.c
rawSHA1_fmt_plug.c
rawSHA1_linkedIn_fmt_plug.c
rawSHA1_ng_fmt_plug.c
rawSHA224_fmt_plug.c
rawSHA256_fmt_plug.c
rawSHA256_ng_fmt_plug.c
rawSHA256_ng_i_fmt_plug.c
rawSHA384_fmt_plug.c
rawSHA512_fmt_plug.c
rawSHA512_ng_fmt_plug.c
rawSHA512_ng_i_fmt_plug.c
ripemd_fmt_plug.c
salted_sha1_fmt_plug.c
sapB_fmt_plug.c
sapG_fmt_plug.c
sha1_gen_fmt_plug.c
siemens-s7_fmt_plug.c
sip_fmt_plug.c
skein_fmt_plug.c
SKEY_fmt_plug.c
snefru_fmt_plug.c
ssh_fmt_plug.c
ssh_ng_fmt_plug.c
ssha512_fmt_plug.c
strip_fmt_plug.c
sunmd5_fmt_plug.c
sxc_fmt_plug.c
SybaseASE_fmt_plug.c
SybasePROP_fmt_plug.c
tcp_md5_fmt_plug.c
tiger_fmt_plug.c
truecrypt_fmt_plug.c
vms_fmt_plug.c
vnc_fmt_plug.c
wbb3_fmt_plug.c
whirlpool_fmt_plug.c
wow_srp_fmt_plug.c
wpapsk_fmt_plug.c
XSHA_fmt_plug.c
XSHA512_fmt_plug.c
zip_fmt_plug.c*

[jfoug]

This comment brought over from a comment on thread #434

I added a dump_stuff to the end of rar's get_salt, and it appears the get_salt is only being called 5 times, so there is some other dupe removal logic that is also working. Rar 'appears' to have the same issue, where it if the same salt was given to rar multiple times, that it would have multiple salts (all the same). BUT jtr is only giving rar the proper amount, but pkzip (from a --list=format-tests replicated 4 times), gets 4x as many salts.

$ ../run/john -form=rar ../in2 -w=../pw
b109105f 5fe0b899 00000000 1c721980 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
42ff7e92 f24fb2f8 00000000 e8721980 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
56ce6de6 ddee17fb 00000000 b0731980 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
c47c5bef 0bbd1e98 01000000 78741980 965f1453 00000000 30000000 00000000 2f000000 00000000 00000000 00000000 30000000 00000000
b4eee1a4 8dc95d12 01000000 b4751980 69a0ebac 00000000 40000000 00000000 2f000000 00000000 00000000 00000000 33000000 00000000
Loaded 5 password hashes with 5 different salts (rar, RAR3 [SHA1 AES 32/32])
No password hashes left to crack (see FAQ)

[jfoug]

This 'might' be a PEBCAK, I need to look more. I can not get pkzip format to salt fail now.

nope, certainly not PEBCAK.

Loaded 50 password hashes with 50 different salts (PKZIP [32/32])

Should be 10 of them (10 hashes, file concatenated 5 times).

[magnumripper]

I just tested rar format (against bug #692) and it appears to be getting 'native' JtR salt dupe removal. Yes, I see that it simply stores the pointer. NOTE, the pointer is pretty deep into the structure. Possibly dupe removal logic only checks so much of the salt, to see if it is a dupe? That in itself 'could' be a bug.

RAR has a cludge in cracker.c line 174. It sets a special "pot check salt size" to a lot lower than the real size. Actually it sets it so the REAL salt is checked but not the pointers and other stuff.

[magnumripper]

RAR has a cludge in cracker.c line 174. It sets a special "pot check salt size" to a lot lower than the real size.

I really dislike changing core functionality if at all possible, but we could add support for this in the format struct. Drop that cludge and announce two salt sizes - one for compare sizeof(saltstruct.salt) and one for salt buffer copy sizeof(saltstruct) (assuming the .salt member comes first in the struct). Normal formats would have them the same size.

[jfoug]

For some formats that would be wonderful. 'most' probably should check the entire salt size. But some (whihc may allocate a buffer and store a unique pointer which of itself is NOT something that can be compared), BUT which does not play into the salt itself, this would be a benefit.

Ok, by what I saw, I would have assumed rar would FAIL dupe salt removal but it worked. Now that you state rar has 'unique help' in core to do this, it makes sense.

Yes, 2 salt sizes (which default to == to each other) might be something helpful for many formats, which need to allocate something to be 'used' but which is not really 'part' of the salt. IT would also allow a lot more pre-computation of 'crap', BUT not have that affect salt dupe removal, as long as it is at the tail end of the salt record, 'after' the part checked. This is not a trivial design issue, and one that developers would have to understand, BUT it is a very useful addition if used properly. My vote is to add this type enhancement.

[magnumripper]

As long as we discuss this on john-dev prior to implementing it, I could live with the core change.

[jfoug]

Issue posted to john-dev. Also while typing that email, a (better) thought popped into my head. Instead of having 2 salt lengths, and keeping a dumb comparator WITHIN john core, why not simply make a 'smart' comparator, by making a salt_equal() method??? That way, the format (which KNOWS it's internal structure), could be in charge of saying, yes this is the same, or no something is different....... The fmt_default_salt_equal() is trivial also. It simply keeps the existing code from today, doing the memcmp call.

For formats that store pointers due to variable sized data, having the format do the compare allow it to simply compare the contents of what those pointers are referencing, vs simply skipping them because the actual bytes that make up the pointer(s) themselves are of no value. Also, the format would 'know' that some data may actually NOT be of any comparator value, but was for instance some pre-computed data. A dumb memcmp would not know this, but the format author would (should).

[jfoug]

Got a script to test, and here are some 'results', showing the zip problems (script named tst, and I run in dir above jtr's run folder)

#!/bin/sh
CNT=`run/john --list=format-tests -format=$1 | wc -l`
TOT=`expr $SSS \* 5`
run/john --list=format-tests -format=$1 | cut -f 3 > in2
run/john --list=format-tests -format=$1 | cut -f 3 >> in2
run/john --list=format-tests -format=$1 | cut -f 3 >> in2
run/john --list=format-tests -format=$1 | cut -f 3 >> in2
run/john --list=format-tests -format=$1 | cut -f 3 >> in2
echo there are $CNT tests for a total of $TOT.  Any salt count more than $CNT is wrong.
echo
run/john -format=$1 in2 --max-run=1
# end of script

$ ./tst zip there are 2 tests for a total of 10. Any salt count more than 2 is wrong.

Loaded 10 password hashes with 10 different salts (ZIP, WinZip [PBKDF2-SHA1 8x SSE2]) Note: This format may emit false positives, so it will keep trying even after finding a possible candidate. Press 'q' or Ctrl-C to abort, almost any other key for status 0g 0:00:00:01 2/3 0g/s 0p/s 3143c/s 3143C/s 123456..bigben Session stopped (max run-time reached)

Now to just make a for loop to do this, and then look over the resultant output for issues like seen above.

[jfoug]

NOTE, if you add a -w=pw (and have a pw file) to the actual john test of the input 2, then zip (and many others) does salt dupe removal correctly ?!?!?!

Changed script to this: run/john -format=$1 in2 --max-run=1 -w=pw

Now ./tst zip

Testing 'zip' format 2 salts (or less) should be seen Loaded 2 password hashes with 2 different salts (ZIP, WinZip [PBKDF2-SHA1 8x SSE2]) Note: This format may emit false positives, so it will keep trying even after finding a possible candidate. Press 'q' or Ctrl-C to abort, almost any other key for status 0g 0:00:00:00 DONE (2014-07-05 08:30) 0g/s 62.50p/s 125.0c/s 125.0C/s x Session completed

So here, we can see that there are only 2 (the correct) hashes being tested. I am getting more and more confused with this.......

[magnumripper]

Oh you must not run batch mode because the dupe rejection is disabled for Single mode!

[jfoug]

why is dupe removal removed for single mode?? This may be the entire issue I am having. If the -w=xxx flag is used, I think all formats are salt deduping (I still need to check things closely). Why on earth would we remove dupe detection?

[magnumripper]

Because if you remove a dupe hash you will lose username and GECOS information. Single mode is all about using username and GECOS fields. This is a core thing. Dupe hashes are same salt anyway so "no" speed is lost.

[jfoug]

I am back to thinking this may be a PEBCAK, and the dupe problem is mostly in the 'if run in single mode'. I have not tested dyna yet, but I see no issues with other formats as long as they are in wordlist mode, or something other than single mode, it seems. I still think salt dupe removal SHOULD be part of single mode. Yes, you may not want to drop candidates, BUT duplicate salts should be dropped. Take for example 2 hashes, but duplicated in a file 1 million times. Even in single mode, that would crawl, 1 million times slower than it should, along with gaining no information.

[magnumripper]

Sure but that is a pretty extreme example. In real world use you'd have a couple of dupe NT hashes which were unsalted anyway. And above all, they share the same bitmap. I do not wish to poke around with Single mode, that is a core thing we should leave as-is IMHO.

For salted formats in real life, any dupe passwords are likely to have different salts.

[magnumripper]

BTW I'm not even sure they are not actually removed. It may be just that the figures shown doesn't account for removed hashes because they are still "used" in terms of GECOS etc.

But this is mostly a john-dev discussion with Solar as the only one who actually knows.

[magnumripper]

I too think this was a pebcak. The only formats I know of that may have the issue are rar and latest zip, and both now have cludges.