ZTEX: support empty salt for md5crypt, sha256crypt, sha512crypt

[AlekseyCherepanov]

I tried md5crypt, sha256crypt and sha512crypt with empty salts and -ztex formats fail with error.

After error, same format is loaded well, but the error happens again. Switching format fails. But rerunning fixes it.

Aborting with Control-C causes john to print Wait..., but then error message is printed again. Next Control-C exits immediately.

@Apingis, can drupal7 have empty salt?

@solardiz, can phpass or bcrypt have empty salt?

$ cat pw-empty-salt
$1$$RmyPVMlhpXjJj8iv4w.Ul.
$5$$ak6faDK9fe20oKsZ7bjHwjN3gWFkUl7Jh1kXkAJFoo3
$6$$.Q5TOqNNyRzKTE5ZY02dsLIOQwkwN0bWooBHB4loBoSN7E68SH7dHJ6SZRyzHvdF0/ybpEO82nrjBd5oI1f3g0

$ john pw-empty-salt --format=md5crypt-ztex
[...]
SN [...] FPGA #1 error: pkt_comm_status=0x80, debug=0x0000
SN [...] error -1 doing r/w of FPGAs (LIBUSB_ERROR_IO)
Found 1 device(s) ZTEX 1.15y
SN: [...] productId: 10.15.0.0 "inouttraffic JtR 1.8.x" busnum:4 devnum:4 
SN [...] FPGA #1 error: pkt_comm_status=0x80, debug=0x0000
SN [...] error -1 doing r/w of FPGAs (LIBUSB_ERROR_IO)
^C
Session aborted

$ john pw-empty-salt --format=sha256crypt-ztex
SN [...]: uploading bitstreams.. ok
SN [...]: device_list_check_bitstreams(): no bitstream or wrong type
no valid ZTEX devices found

$ john pw-empty-salt --format=sha256crypt-ztex
SN [...]: uploading bitstreams.. ok
ZTEX [...] bus:4 dev:4 Frequency:175 175 175 175 
[...]
SN [...] FPGA #1 error: pkt_comm_status=0x80, debug=0x0000
SN [...] error -1 doing r/w of FPGAs (LIBUSB_ERROR_IO)

[...]
$ john pw-empty-salt --format=sha512crypt-ztex
[...]
SN [...] FPGA #1 error: pkt_comm_status=0x80, debug=0x0000
SN [...] error -1 doing r/w of FPGAs (LIBUSB_ERROR_IO)

$ john pw-empty-salt --format=md5crypt
[...]
123456           (?)
[...]
$ john pw-empty-salt --format=sha256crypt
[...]
123456           (?)
[...]
$ john pw-empty-salt --format=sha512crypt
[...]
123456           (?)
[...]

$ john --list=build-info
Version: 1.8.0.16-jumbo-1-bleeding-e9f0f93 2019-03-29 20:46:07 +0200
[...]

$ ldd JohnTheRipper/run/john
[...]
libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0
[...]
$ apt-file search /lib/x86_64-linux-gnu/libusb-1.0.so.0
libusb-1.0-0: /lib/x86_64-linux-gnu/libusb-1.0.so.0
[...]

$ apt-cache show libusb-1.0-0
Package: libusb-1.0-0
Source: libusb-1.0
Version: 2:1.0.19-1
[...]

[Apingis]

Empty salt (salt_len=0) is not supported. Never seen such in test vectors or elsewhere. I think I'll correct in the next version.

SN [...] FPGA #1 error: pkt_comm_status=0x80 arises because it doesn't accept salt_len=0, treats as error in input packet.

Pressing Ctrl-C once waits until it finishes crypt_all() which doesn't happen because boards return errors.

drupal7 hash format has no separator between salt and hash e.g. $S$CFURCPa.k6FAEbJPgejaW4nijv7rYgGc4dUJtChQtV4KLJTPTC/u and ciphertext length is fixed so John would not recognize drupal7 hashes unless they have 8-byte salt. To be absolutely sure on the issue, one has to take a look into the code of the application that generates hashes.

[AlekseyCherepanov]

I made the hashes using python's crypt module from libpython2.7-stdlib Debian package.

>>> import crypt
>>> help(crypt.crypt)
Help on built-in function crypt in module crypt:

crypt(...)
    crypt(word, salt) -> string
    word will usually be a user's password. salt is a 2-character string
    which will be used to select one of 4096 variations of DES. The characters
    in salt must be either ".", "/", or an alphanumeric character. Returns
    the hashed password as a string, which will be composed of characters from
    the same alphabet as the salt.

>>> crypt.crypt('123456', '$1$')
'$1$$RmyPVMlhpXjJj8iv4w.Ul.'
>>> crypt.crypt('123456', '$5$')
'$5$$ak6faDK9fe20oKsZ7bjHwjN3gWFkUl7Jh1kXkAJFoo3'
>>> crypt.crypt('123456', '$6$')
'$6$$.Q5TOqNNyRzKTE5ZY02dsLIOQwkwN0bWooBHB4loBoSN7E68SH7dHJ6SZRyzHvdF0/ybpEO82nrjBd5oI1f3g0'

[AlekseyCherepanov]

C variant allows such use too:

$ echo 'main() { puts(crypt("123456", "$1$")); }' | tcc -lcrypt -run -
$1$$RmyPVMlhpXjJj8iv4w.Ul.

[magnumripper]

I'm pretty sure I've seen $1$$ in the wild

[solardiz]

I also have seen $1$$ in the wild. We should support empty salt for md5crypt, sha256crypt, and sha512crypt.

bcrypt, phpass, and Drupal7 have no separator character between salt and hash. They use fixed length salts.

[solardiz]

We do have empty salt test vectors for md5crypt, but not for sha-crypt. We should probably add such test vectors for the latter, for CPU and OpenCL as well.

[solar@super src]$ fgrep '"$1$$' *.c
MD5_fmt.c:  {"$1$$qRPK7m23GJusamGpoGLby/", ""},
MD5_fmt.c:  {"$1$$AuJCr07mI7DSew03TmBIv/", "no salt"},
c3_fmt.c:           {"$1$$qRPK7m23GJusamGpoGLby/", ""},
opencl_md5crypt_fmt_plug.c: {"$1$$qRPK7m23GJusamGpoGLby/", ""},
opencl_md5crypt_fmt_plug.c: {"$1$$AuJCr07mI7DSew03TmBIv/", "no salt"},
[solar@super src]$ fgrep '"$5$$' *.c
[solar@super src]$ fgrep '"$6$$' *.c
[solar@super src]$ 

[solardiz]

Also, while the bitstreams don't accept empty salts, perhaps the host-side code should know not to load such hashes? Maybe print warnings from valid when it'd return 0 for that reason? Or would adding this be a waste of effort, compared to fixing the underlying issue in the hardware designs?

[Apingis]

I'm adding host-side code not to load such hashes, print warnings. Fixing hardware design wouldn't be a big issue, however it would take time to generate new bitstreams.

[solardiz]

@AlekseyCherepanov Please test the workaround introduced with #3742. Thanks.

[AlekseyCherepanov]

New valid() functions return 0 for salts with hash and print a warning. Warning is printed for every hash, so multiple hashes of same type with empty salt mean same warning printed multiple times.

[AlekseyCherepanov]

valid() is called parsing .pot, so warning is emitted for it too:

$ echo '$1$$RmyPVMlhpXjJj8iv4w.Ul.:123456' > t.pot
$ echo '$1$asdf$s3o4V7L52cI4MFl79jdCE/' > t.pw
$ john --pot=t.pot t.pw
[...]
Loaded 1 password hash (md5crypt-ztex, crypt(3) $1$ [md5crypt ZTEX])
Warning: ZTEX: md5crypt hash with salt_length=0 skipped.
[...]
123456           (?)
[...]
$ cat t.pot
$1$$RmyPVMlhpXjJj8iv4w.Ul.:123456
$1$asdf$s3o4V7L52cI4MFl79jdCE/:123456

[Apingis]

I see. The host-side workaround looks clumsy both in terms of overall system design and maybe in terms of quick implementation.

Overally the issue seems to be substantial, I can imagine developers of e.g. authentication for web-services, using hashing improperly w/o a salt, as a result such hashes may appear as a subject for John.

[AlekseyCherepanov]

Some sed magic to print some valid() functions to see implementations of warnings:

$ sed -ne '/.*int [^ ]*valid(/,/^}/ p' JohnTheRipper/src/*.c

JohnTheRipper/src/keepass_fmt_plug.c:

int keepass_valid(char *ciphertext, struct fmt_main *self)
{
[...]
        if (contentsize > MAX_CONT_SIZE) {
            static int warned;

            if (!ldr_in_pot && warned < contentsize) {
                fprintf(stderr,
                        "%s: Input rejected due to larger size than compile-time limit.\n"
                        "Bump MAX_CONT_SIZE in keepass_common.h to >= 0x%x, and rebuild\n",
                        self->params.label, contentsize);
                warned = contentsize;
            }
            goto err;
        }
[...]
}

[magnumripper]

Yes, ldr_in_pot is the traffic light to watch here - simply don't print warnings from valid() when that one is true. Also, most of this stuff should better also depend on john_main_process to avoid multiple output for MPI processes, but that might not be much of an issue for ZTEX formats: I'm not sure the ztex formats has all MPI/node/fork logic that OpenCL has?

[solardiz]

I'm not sure the ztex formats has all MPI/node/fork logic that OpenCL has?

I think ZTEX will work with --node fine. It won't work with --fork in any reasonable way - all the forked processes will try to use the same devices, and will clash. I don't know whether it'll work with MPI or not.

[magnumripper]

Then chances are good they will work just fine under MPI as-is. The "special" MPI logic in OpenCL is all about multi-GPU machines using one process per device.