Lots of formats are missing a salt_hash function

[magnumripper]

Many of them don't need one, but some do. See also #809

[magnumripper]

Using this temporary patch:

diff --git a/src/formats.c b/src/formats.c
index 73efe6da3..e3843b720 100644
--- a/src/formats.c
+++ b/src/formats.c
@@ -463,6 +463,10 @@ static char *fmt_self_test_body(struct fmt_main *format,
    if (format->methods.valid("*", format))
        return "valid";

+   if (format->methods.salt_hash == fmt_default_salt_hash &&
+       format->params.binary_size && format->params.salt_size)
+       puts("Warning: missing salt_hash()");
+
    if (format->methods.source != fmt_default_source &&
        format->params.salt_size != 0)
        return "source method only allowed for unsalted formats";

...lists these with -test=0 -form=cpu

Testing: AFS, Kerberos AFS [DES 48/64 4K]... Warning: missing salt_hash()
Testing: aix-ssha1, AIX LPA {ssha1} [PBKDF2-SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: aix-ssha256, AIX LPA {ssha256} [PBKDF2-SHA256 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: aix-ssha512, AIX LPA {ssha512} [PBKDF2-SHA512 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: ansible, Ansible Vault [PBKDF2-SHA256 HMAC-256 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: as400-des, AS/400 DES [DES 32/64]... Warning: missing salt_hash()
Testing: AzureAD [PBKDF2-SHA256 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: Blackberry-ES10 (101x) [SHA-512 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: chap, iSCSI CHAP authentication / EAP-MD5 [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: Django (x10000) [PBKDF2-SHA256 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: django-scrypt [Salsa20/8 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: dmd5, DIGEST-MD5 C/R [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: eCryptfs (65536x) [SHA512 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: eigrp, EIGRP MD5 / HMAC-SHA-256 authentication [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: EPiServer [SHA1/SHA256 32/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: ethereum, Ethereum Wallet [PBKDF2-SHA256/scrypt Keccak 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: gost, GOST R 34.11-94 [64/64]... (8xOMP) Warning: missing salt_hash()
Testing: hdaa, HTTP Digest access authentication [MD5 32/64]... Warning: missing salt_hash()
Testing: hsrp, "MD5 authentication" HSRP, HSRPv2, VRRP, GLBP [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: IKE, PSK [HMAC MD5/SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: keystore, Java KeyStore [SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: known_hosts, HashKnownHosts HMAC-SHA1 [SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: krb5pa-sha1, Kerberos 5 AS-REQ Pre-Auth etype 17/18 [PBKDF2-SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: krb5-17, Kerberos 5 DB etype 17 [DES / PBKDF2-SHA1 32/64 AES]... (8xOMP) Warning: missing salt_hash()
Testing: krb5-18, Kerberos 5 DB etype 18 [DES / PBKDF2-SHA1 32/64 AES]... (8xOMP) Warning: missing salt_hash()
Testing: krb5-3, Kerberos 5 DB etype 3 [DES / PBKDF2-SHA1 32/64 AES]... (8xOMP) Warning: missing salt_hash()
Testing: lp, LastPass offline [PBKDF2-SHA256 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: lpcli, LastPass CLI [PBKDF2-SHA256 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: MongoDB, system / network [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: scram [SCRAM PBKDF2-SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: Mozilla, Mozilla key3.db [SHA1 3DES 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: mysqlna, MySQL Network Authentication [SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: net-ah, IPsec AH HMAC-MD5-96 [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: net-md5, "Keyed MD5" RIPv2, OSPF, BGP, SNMPv2 [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: net-sha1, "Keyed SHA1" BFD [SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: nk, Nuked-Klan CMS [SHA1 MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: ODF, OpenDocument Star/Libre/OpenOffice [PBKDF2-SHA1 BF/AES 32/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: Office, 2007/2010/2013 [SHA1 32/64 / SHA512 64/64 OpenSSL AES]... (8xOMP) Warning: missing salt_hash()
Testing: OpenBSD-SoftRAID (8192 iterations) [PBKDF2-SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: Oracle12C [PBKDF2-SHA512 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: ospf, OSPF / IS-IS [HMAC-SHA-X 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: PBKDF2-HMAC-MD4 [PBKDF2-MD4 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: PBKDF2-HMAC-MD5 [PBKDF2-MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: PBKDF2-HMAC-SHA1 [PBKDF2-SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: PBKDF2-HMAC-SHA256 [PBKDF2-SHA256 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: PBKDF2-HMAC-SHA512, GRUB2 / OS X 10.8+ [PBKDF2-SHA512 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: pfx [PKCS12 PBE (.pfx, .p12) (SHA-1 to SHA-512) 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: pgpdisk [PGP Disk / Virtual Disk SHA1 64]... (8xOMP) Warning: missing salt_hash()
Testing: pgpsda [PGP SDA SHA1 64]... (8xOMP) Warning: missing salt_hash()
Testing: pomelo [POMELO 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: postgres, PostgreSQL C/R [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: pwsafe, Password Safe [SHA256 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: RACF [DES 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: RAKP, IPMI 2.0 RAKP (RMCP+) [HMAC-SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: RAR5 [PBKDF2-SHA256 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: rsvp, HMAC-MD5 / HMAC-SHA1, RSVP, IS-IS, OMAPI, RNDC, TSIG [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: SIP [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: skey, S/Key [MD4/MD5/SHA1/RMD160 32/64]... Warning: missing salt_hash()
Testing: LastPass, sniffed sessions [PBKDF2-SHA256 AES 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: sspr, NetIQ SSPR / Adobe AEM [MD5/SHA1/SHA256/SHA512 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: Sybase-PROP [salted FEAL-8 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: tcp-md5, TCP MD5 Signatures, BGP, MSDP [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: OpenVMS, Purdy [32/64]... (8xOMP) Warning: missing salt_hash()
Testing: VNC [DES 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: vtp, "MD5 based authentication" VTP [MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: wbb3, WoltLab BB3 [SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: xmpp-scram [XMPP SCRAM PBKDF2-SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: HMAC-MD5 [password is key, MD5 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: HMAC-SHA1 [password is key, SHA1 32/64]... (8xOMP) Warning: missing salt_hash()
Testing: HMAC-SHA224 [password is key, SHA224 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: HMAC-SHA256 [password is key, SHA256 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: HMAC-SHA384 [password is key, SHA384 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()
Testing: HMAC-SHA512 [password is key, SHA512 64/64 OpenSSL]... (8xOMP) Warning: missing salt_hash()

The vast majority of these don't really need a salt hash function. The aix-ssha* formats probably do, and perhaps the as400-des format. And what more? GOST perhaps? DMD5? RACF?

[jfoug]

I just did a bit of work with single mode bugs (large data sets, etc). I tried hmac-md5, and it would not load (very very slow). That format has another possible big bug in it. But it certainly needs salt hash.

I then tried hmac-sha1. Same problem with salt hash. I started a load going with 250k hashes built from pass_gen. After an hour, I gave up. I simply added a trivial salt_hash (returned first 4 bytes of salt % (HASH_SIZE+1), set salt align to 4. The 250k now load in a couple seconds.

I would assume that all of these could be done, in a very trivial manner, and SHOULD be. It does no harm at all having that hashing there. It simply makes the load happen MUCH faster.

As for the warning, it should be there full time, once we get salt hash code inserted into each format, UNLESS there is a format where we can not do so.

[magnumripper]

It's a pity we can't "see" params.salt_size from fmt_default_salt_hash(). If we could, the default function could be some simple hash function over the whole size, and it would be perfectly fine for most formats.

[magnumripper]

...but we could make a standard (but not default) salt_hash() function and put in salt_hash_std.h and just include it from many formats. It could use SALT_SIZE.

[jfoug]

Yep, that was the plan ;)

Do we have functions which do 'default' but normal processing within formats.[ch] Note, the fmt_default_salt_hash is a dummy 'null' function, which probably should not have it's functionality changed. But for this, we may want something like fmt_common_salt_hash() It would require 4 byte aligned salt, and a salt that is at least 4 bytes long. It would simply take the first 32 bits mod the hash size. It is FAR from a perfect function, BUT it makes a tremendous difference (even though it 'could' be done better) when the data set is large.

[magnumripper]

It would require 4 byte aligned salt, and a salt that is at least 4 bytes long. It would simply take the first 32 bits mod the hash size. It is FAR from a perfect function, BUT it makes a tremendous difference (even though it 'could' be done better) when the data set is large.

If we do one shared function for 200+ formats, why on earth wouldn't we do a real hash function that use all salt data according to SALT_SIZE?! Also, it should not require aligned data.

[AlekseyCherepanov]

I'll add a subproblem here: some formats (gpg, keystore, 7z) use fmt_default_salt_hash together with FMT_DYNA_SALT while they could use fmt_default_dyna_salt_hash.

[AlekseyCherepanov]

A set of predefined functions for common sizes of salts could be pre-populated. Then fmt_default_salt_hash could be replaced with a function for correct size. If there is not a precise match then a function for closest smaller size could be used.

Or a global variable for self could be used. As far as I understand, salt_hash is used only with a format selected for cracking. So a var should be ok. (For comparison calls to valid()s are intermixed across formats to detect alternatives.)

[AlekseyCherepanov]

I guess fmt_default_salt_hash can be replaced safely if salt_compare is not set. Formats with salt_compare might do something smart, so a deeper review would be needed.

Some formats store variable length salt using fixed length and provide neither salt_compare nor salt_hash. Some of them would not fill unused space with zeroes. Then without salt_compare duplicate line in input file might be loaded twice as different hashes. It would be a bug. I actually tried it including a pointer into salt working on #5120 (it has salt_hash though). But without use of pointers, contents of format's static buffer would be rather stable and will not uncover the bug easily. So a loading sequence like short salt, long salt, the same short salt is needed. From theoretical pov I don't see reasons for proper salt_hash to make things worse.

I guess format should 1) have true fixed salt, 2) use dyna_salt, or 3) have variable length salt and fill unused space with zeroes. Then there would not be reasons for custom salt_compare and salt_hash at all: just compare with memcmp and hash with xxhash (maybe 64-bit variant of xxhash truncated to 32-bit would be faster on 64-bit machines).

[solardiz]

Thank you @AlekseyCherepanov. What's the reason/trigger for your renewed interest in this specific issue? Perhaps you ran into it for some specific format/usage?

Re-reading past comments, it appears a quick fix that would cover most cases would be to have a second/alternative default salt hash function that would use e.g. first 4 bytes, and use that for formats where salt size is at least that.

Please feel free to contribute a fix/improvement.

Regarding generic/third-party hash functions to consider for larger inputs, see #4350.

[AlekseyCherepanov]

I'm writing a generic tracer for formats. It makes a copy of methods and substitutes them with its own methods that proxy calls to the copied methods.

Unusual configuration of salt_hash in gpg, keystore, 7z led me to this issue. So I don't have a problem requiring a quick fix. Thanks!

[AlekseyCherepanov]

I guess format should 1) ... 2) ... 3) ...

Well, I can imagine that format could precompute something in salt() expanding result greatly during loading while comparison of a small part would be enough. Then custom salt_hash and salt_compare might be beneficial even for truly fixed size salts.

BTW is it ok to store additional precomputed data in salt?