Formats with NULL pointers for (some) binary_hash and get_hash methods

[frank-dittrich]

This is for a non-GPU build (64bit linux)

$ ./john --list=format-methods:binary_hash |grep -E "(^Methods|NULL pointer)" | grep -B 1 NULL     
Methods overridden for:   scrypt [Salsa20/8 128/128 AVX] 
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
--
Methods overridden for:   AFS [DES 48/64 4K] Kerberos AFS
        binary_hash[3]()  (NULL pointer)
        binary_hash[4]()  (NULL pointer)
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[3]()  (NULL pointer)
        get_hash[4]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
--
Methods overridden for:   dynamic_19 [Cisco PIX (MD5) 128/128 AVX 480x4x3] 
        binary_hash[6]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
Methods overridden for:   dynamic_20 [Cisco ASA (MD5 salted) 128/128 AVX 480x4x3] 
        binary_hash[6]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
--
Methods overridden for:   MSCHAPv2 [MD4 DES (ESS MD5) 128/128 AVX 12x] C/R
        binary_hash[4]()  (NULL pointer)
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[4]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
--
Methods overridden for:   netntlm [MD4 DES (ESS MD5) 128/128 AVX 12x] NTLMv1 C/R
        binary_hash[4]()  (NULL pointer)
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[4]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
--
Methods overridden for:   oracle [DES 32/64] Oracle 10
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
--
Methods overridden for:   PKZIP [32/64] 
        binary_hash[1]()  (NULL pointer)
        binary_hash[2]()  (NULL pointer)
        binary_hash[3]()  (NULL pointer)
        binary_hash[4]()  (NULL pointer)
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[1]()  (NULL pointer)
        get_hash[2]()  (NULL pointer)
        get_hash[3]()  (NULL pointer)
        get_hash[4]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
--
Methods overridden for:   tc_ripemd160 [32/64] TrueCrypt RIPEMD160 AES256_XTS
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
Methods overridden for:   tc_sha512 [64/64] TrueCrypt SHA512 AES256_XTS
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
Methods overridden for:   tc_whirlpool [64/64] TrueCrypt WHIRLPOOL AES256_XTS
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)
--
Methods overridden for:   crypt [?/64] generic crypt(3)
        binary_hash[5]()  (NULL pointer)
        binary_hash[6]()  (NULL pointer)
        get_hash[5]()  (NULL pointer)
        get_hash[6]()  (NULL pointer)

[magnumripper]

mschapv2 and ntlmv1 (non naive) both have a too small binary size to use any larger functions iirc. Some/many of the others may have similar limitations. Again, pkzip has a binary size of 0 so that's a red herring. Still, it should use the default functions just for not getting hit by this kind of tests.

[magnumripper]

Actually only Oracle really needed a fix (pkzip and TC were already fixed).

[frank-dittrich]

If I run

./john --list=format-methods|LC_ALL=C grep -E "(Methods overridden|NULL pointer)"|grep -B 1 "NULL pointer" |grep -v -- "--"

on bleeding-jumbo, the list has gotten longer than on Oct 7. And these formats don't even have binary_hash[0]() / get_hash[0]():

./john --list=format-methods|LC_ALL=C grep -E "(Methods overridden|\[0\]\(\)  \(NULL pointer)"|grep -B 1 "NULL pointer" |grep -v -- "--"      
Methods overridden for:   asa-md5 [?] Cisco ASA
        binary_hash[0]()  (NULL pointer)
        get_hash[0]()  (NULL pointer)
Methods overridden for:   FormSpring [?] 
        binary_hash[0]()  (NULL pointer)
        get_hash[0]()  (NULL pointer)
Methods overridden for:   MediaWiki [?] 
        binary_hash[0]()  (NULL pointer)
        get_hash[0]()  (NULL pointer)
Methods overridden for:   osc [?] osCommerce
        binary_hash[0]()  (NULL pointer)
        get_hash[0]()  (NULL pointer)
Methods overridden for:   phpass [?] 
        binary_hash[0]()  (NULL pointer)
        get_hash[0]()  (NULL pointer)
Methods overridden for:   PHPS [?] 
        binary_hash[0]()  (NULL pointer)
        get_hash[0]()  (NULL pointer)
Methods overridden for:   pix-md5 [?] Cisco PIX
        binary_hash[0]()  (NULL pointer)
        get_hash[0]()  (NULL pointer)

[frank-dittrich]

For some reason, opencl-pwsafe has just fmt_default_binary_hash and fmt_default_get_hash, while pwsafe (CPU format) has

                        fmt_default_binary_hash_0,
                        fmt_default_binary_hash_1,
                        fmt_default_binary_hash_2,
                        fmt_default_binary_hash_3,
                        fmt_default_binary_hash_4,
                        fmt_default_binary_hash_5,
                        fmt_default_binary_hash_6

and

                        get_hash_0,
                        get_hash_1,
                        get_hash_2,
                        get_hash_3,
                        get_hash_4,
                        get_hash_5,
                        get_hash_6

[frank-dittrich]

pwsafe-cuda uses the same functions as pwsafe-opencl

[magnumripper]

on bleeding-jumbo, the list has gotten longer than on Oct 7. And these formats don't even have binary_hash[0]() / get_hash[0]():

Aren't all of these thin dynamic formats? Things might not be quite like they seem to be.

[magnumripper]

The GPU pwsafe formats has a binary size of 0, which kind of explains it. BUT I really wonder why they have zero while the CPU format has 32.

[frank-dittrich]

| Aren't all of these thin dynamic formats? Things might not be quite like they seem to.

Yes, those formats which "don't have" binary_hash[0]() and get_hash[0]() are indeed thin dynamic formats. But net-md5 and net-sha1 are thin dynamic formasts as well. And these two do have 7 binary_hash functions and 7 get_hash functions each. I am confused.

[magnumripper]

Not sure but IIRC those two are a mix of thick and thin formats. @jfoug will fill in the blanks for us.

[frank-dittrich]

If asa-md5 is linked to dynamic_20, shouldn't

./john --list=format-methods --format=dynamic_20

and

./john --list=format-methods --format=asa-md5

print the same methods? Do we need to somehow adjust the --list=format-methods output?

[jfoug]

Net md5 sha1 are hybrid. Sometimes think sometime thin. They are really think formats that can also use some dynamic stuff..

[jfoug]

print the same methods? Do we need to somehow adjust the --list=format-methods output?

I do not know logic behind --list=format-methods, I did not write it, but i made this change to init() within asamd5:

  static void init(struct fmt_main *self)
  {
+   printf ("init() called\n");
    get_ptr();

Then ran the --list=show-formats:

$ ../run/john --list=format-methods --form=asa-md5
Methods overridden for:   asa-md5 [?] Cisco ASA
        init()
        prepare()
        valid()
        split()
        binary()
        salt()
        source()
                binary_hash[0]()  (NULL pointer)
                binary_hash[1]()  (NULL pointer)
                binary_hash[2]()  (NULL pointer)
                binary_hash[3]()  (NULL pointer)
                binary_hash[4]()  (NULL pointer)
                binary_hash[5]()  (NULL pointer)
                binary_hash[6]()  (NULL pointer)
        salt_hash()
        set_salt()
        set_key()
        get_key()
        clear_keys()
                get_hash[0]()  (NULL pointer)
                get_hash[1]()  (NULL pointer)
                get_hash[2]()  (NULL pointer)
                get_hash[3]()  (NULL pointer)
                get_hash[4]()  (NULL pointer)
                get_hash[5]()  (NULL pointer)
                get_hash[6]()  (NULL pointer)
        crypt_all()
        cmp_all()
        cmp_one()
        cmp_exact()

Init() is never called. That is why the func pointers may not look the same. What you are getting is only the thin side of the format, not the format after it has properly linked itself to dyna.

[magnumripper]

So it needs init() just like some other --list stuff

        if (!strcasecmp(options.listconf, "format-details")) {
                struct fmt_main *format;

#if HAVE_OPENCL
                /* This will make the majority of OpenCL formats
                   also do "quick" run. But if LWS or
                   GWS was already set, we do not overwrite. */
                setenv("LWS", "7", 0);
                setenv("GWS", "49", 0);
                setenv("BLOCKS", "7", 0);
                setenv("THREADS", "7", 0);
#endif
                format = fmt_list;
                do {
                        int ntests = 0;

                        fmt_init(format);       /* required for --encoding support */

In this case we might get away with only doing it if (formats->params.flags & FMT_DYNAMIC) or something like that

[frank-dittrich]

@jfoug

| I do not know logic behind --list=format-methods, I did not write it

You didn't? git show75a95e287000f40cea345bbbb7f11dac6c469385` suggests something else. OK, commit 75a95e287000f40cea345bbbb7f11dac6c469385 is 2.5 years old, enough time to forget this. And all the --list= logic was still in john.c, and magnium later moved it to listconf.c.

[frank-dittrich]

I added fmt_init() and fmt_done() for --list=format-methods in commit 4bbc28468c83137a35da148f6ed18f5396c0c492, to fix the output for thin formats.

[magnumripper]

4bbc284 repeats mistakes we've done before: Try an opencl format and it will be hugely delayed by the init(). See the example code I posted above. Or better, only do the init if format name does not contain "-opencl" or "-cuda".

[frank-dittrich]

OK, will fix, Thanks

[frank-dittrich]

Done, commit fe3b4d48908d9b74a60b2f6162ef45f2f534eb95

[jfoug]

I do not know logic behind --list=format-methods, I did not write it

You didn't? git show75a95e287000f40cea345bbbb7f11dac6c469385` suggests something else. OK, commit 75a95e287000f40cea345bbbb7f11dac6c469385 is 2.5 years old, enough time to forget

I hate having some computer program with a memory like an elephant, and me, losing mine.

[magnumripper]

So can we close this now?

[frank-dittrich]

Yep.