15 DES-using formats fail self-test with OpenSSL 3.0.0

[DontBreakAlex]

:1st_place_medal::thumbsup:

Self test fails for opeenssh private key:

➜  run git:(bleeding-jumbo) ✗ ssh-keygen                           
Generating public/private rsa key pair.
Enter file in which to save the key (/home/alexandre/.ssh/id_rsa): test
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in test
Your public key has been saved in test.pub
The key fingerprint is:
SHA256:eo9jvwao/+efJ7BIRvDwsWw4bJgyKhW/H+4H8lwl4s0 alexandre@DENGROUS
The key's randomart image is:
+---[RSA 3072]----+
|                 |
|  .   o .        |
|   o + B o       |
|  + + * O .      |
| o o + OSo       |
|o   o =.E .      |
|.    B.*.o o     |
|    . =.=o+ ...  |
|     oo+o*=ooo   |
+----[SHA256]-----+
➜  run git:(bleeding-jumbo) ✗ ssh-keygen -f test -l                
3072 SHA256:eo9jvwao/+efJ7BIRvDwsWw4bJgyKhW/H+4H8lwl4s0 alexandre@DENGROUS (RSA)
➜  run git:(bleeding-jumbo) ✗ ./ssh2john.py test > test.hash
➜  run git:(bleeding-jumbo) ✗ ./john test.hash --mask='?d?d?d'     
[ssh-opencl] cipher value of 6 is not yet supported with OpenCL!
Using default input encoding: UTF-8
Loaded 1 password hash (SSH, SSH private key [RSA/DSA/EC/OPENSSH 32/64])
Cost 1 (KDF/cipher [0=MD5/AES 1=MD5/3DES 2=Bcrypt/AES]) is 2 for all loaded hashes
Cost 2 (iteration count) is 16 for all loaded hashes
Will run 32 OpenMP threads
Self test failed (cmp_all(2))

The passphrase is 12345

Output of ./john --list=build-info:

➜  run git:(bleeding-jumbo) ✗ ./john --list=build-info
Version: 1.9.0-jumbo-1+bleeding-5a19e3d66 2021-09-24 04:35:26 +0200
Build: linux-gnu 64-bit x86_64 AVX2 AC OMP
SIMD: AVX2, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
CPU tests: AVX2
$JOHN is ./
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 10.3.0
GNU libc version: 2.31 (loaded: 2.31)
OpenCL headers version: 1.2
Crypto library: OpenSSL
OpenSSL library version: 030000000
OpenSSL 3.0.0-dev xx XXX xxxx
GMP library version: 6.2.0
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
times(2) sysconf(_SC_CLK_TCK) is 100
Using times(2) for timers, resolution 10 ms
HR timer: clock_gettime(), latency 40 ns
Total physical host memory: 32089 MiB
Available physical host memory: 27062 MiB
Terminal locale string: en_US.UTF-8
Parsed terminal locale: UTF-8

OpenSSH version:

➜  run git:(bleeding-jumbo) ✗ ssh -V
OpenSSH_8.2p1 Ubuntu-4ubuntu0.3, OpenSSL 1.1.1f  31 Mar 2020

My CPU is a AMD Ryzen 9 3950X My GPU is a NVIDIA GeForce GTX 1080 Ti

Files used: files.zip

John is built from source on Ubuntu 20.04 with kernel 5.4.0-86-generic

Thanks

[claudioandre-br]

This is interesting because you are using a CPU format. Well, And it seems to be ok here and inside CI.

$ john test.hash --mask='?d?d?d'
[ssh-opencl] cipher value of 6 is not yet supported with OpenCL!
Using default input encoding: UTF-8
Loaded 1 password hash (SSH, SSH private key [RSA/DSA/EC/OPENSSH 32/64])
Cost 1 (KDF/cipher [0=MD5/AES 1=MD5/3DES 2=Bcrypt/AES]) is 2 for all loaded hashes
Cost 2 (iteration count) is 16 for all loaded hashes
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
123              (test)     
1g 0:00:00:07 DONE (2021-09-24 13:49) 0.1256g/s 48.24p/s 48.24c/s 48.24C/s 123..363
Use the "--show" option to display all of the cracked passwords reliably
Session completed. 

Testing: SSH, SSH private key [RSA/DSA/EC/OPENSSH 32/64]... (2xOMP) PASS

Could you please try to narrow the culprit?

[DontBreakAlex]

I tried on my home server running Debian 10 with a AMD Ryzen 7 1700X CPU, and it works fine. I've tried lowering my RAM and CPU frequency on my workstation, but it does not fix the issue. I did a memtest not too long ago, so I don't think this is an hardware issue.

Do you have an idea about what I could try to narrow down the issue ? If someone with a 3950X reads this, can you try to reproduce the issue please ?

[solardiz]

Do you have an idea about what I could try to narrow down the issue ?

Compare the gcc versions. Try the same gcc version on both systems.

[solardiz]

Also compare OpenSSL versions. I notice the failing one is reported as OpenSSL 3.0.0-dev xx XXX xxxx - it's not too surprising that a development version is unreliable.

Also try prefixing the ./john test.hash --mask='?d?d?d' (on the system where it fails) with OMP_NUM_THREADS=1, so:

OMP_NUM_THREADS=1 ./john test.hash --mask='?d?d?d'

[DontBreakAlex]

Great catch, this was the issue. I rebuilt john with OpenSSL 1.1.1f and it works. Tanks !

[solardiz]

@DontBreakAlex Where did that "OpenSSL 3.0.0-dev xx XXX xxxx" come from - was it a distro package, or was it something you built yourself? Can you test whether the problem is still present with the 3.0.0 release (yes, it was in fact released a couple of weeks ago)? If OpenSSL still has this bug in their codebase, we could want to report it to them. Thanks!

[DontBreakAlex]

@solardiz It was not a OpenSSL release, I had built it straight from git, and had forgotten that I had done so. I just tried with OpenSSL 3.0.0 7 sep 2021 (archive from OpenSSL's website) and it fails:

➜  run git:(bleeding-jumbo) ✗ ./john --list=build-info
Version: 1.9.0-jumbo-1+bleeding-5a19e3d66 2021-09-24 04:35:26 +0200
Build: linux-gnu 64-bit x86_64 AVX2 AC OMP
SIMD: AVX2, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
CPU tests: AVX2
$JOHN is ../run/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 10.3.0
GNU libc version: 2.31 (loaded: 2.31)
OpenCL headers version: 1.2
Crypto library: OpenSSL
OpenSSL library version: 030000000
OpenSSL 3.0.0 7 sep 2021
GMP library version: 6.2.0
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
times(2) sysconf(_SC_CLK_TCK) is 100
Using times(2) for timers, resolution 10 ms
HR timer: clock_gettime(), latency 40 ns
Total physical host memory: 32089 MiB
Available physical host memory: 22667 MiB
Terminal locale string: en_US.UTF-8
Parsed terminal locale: UTF-8
➜  run git:(bleeding-jumbo) ✗ ./john test.hash --mask='?d?d?d?d?d'
[ssh-opencl] cipher value of 6 is not yet supported with OpenCL!
Using default input encoding: UTF-8
Loaded 1 password hash (SSH, SSH private key [RSA/DSA/EC/OPENSSH 32/64])
Cost 1 (KDF/cipher [0=MD5/AES 1=MD5/3DES 2=Bcrypt/AES]) is 2 for all loaded hashes
Cost 2 (iteration count) is 16 for all loaded hashes
Will run 32 OpenMP threads
Self test failed (cmp_all(2)

I also tried on a clean VM to make sure that my install wasn't broken, and it fails there too.

I could be that OpenSSH generated the key with OpenSSL 1.1, and so using OpenSSL 3.0 with john breaks stuff, but this I'm only guessing. I'll try to build OpenSSH with OpenSSL 3.0 tomorrow, if it works.

[solardiz]

@DontBreakAlex Thank you for helping us figure this out. I've reopened the issue and assigned it to a milestone because this is something we'll want to at least attempt fixing or working around in OpenSSL or JtR.

[solardiz]

So this was failing at cmp_all(2), thus (given our current tests) on the second test vector. That one uses MD5 and 3DES. I guess the failure is related to deprecation of (at least one of) these algorithms in latest OpenSSL. Do the APIs we use possibly tell us these algorithms are no longer supported? In that case, I guess we'd have to exclude such test vectors and (unfortunately!) refuse to load such "hashes" (or implement our own support not relying on OpenSSL, or bundle pieces from sufficiently old OpenSSL in JtR). In fact, we can also have JtR determine the same from where the self-tests fail and act accordingly (smarter than refusing to work for any SSH "hashes" at all), but that would be more of a hack.

$ ./john -list=format-tests -format=ssh | head -2 | tail -1 | cut -f3 -d'      ' > pw
$ ./john pw
[...]
Loaded 1 password hash (SSH, SSH private key [RSA/DSA/EC/OPENSSH 32/64])
Cost 1 (KDF/cipher [0=MD5/AES 1=MD5/3DES 2=Bcrypt/AES]) is 1 for all loaded hashes
Cost 2 (iteration count) is 2 for all loaded hashes

Curiously, apparently self-tests for the first test vector passed, and that one uses MD5 and AES. So perhaps it's only deprecation of (3)DES that's at play here, not of MD5.

$ ./john -list=format-tests -format=ssh | head -1 | cut -f3 -d'        ' > pw
$ ./john pw
[...]
Loaded 1 password hash (SSH, SSH private key [RSA/DSA/EC/OPENSSH 32/64])
Cost 1 (KDF/cipher [0=MD5/AES 1=MD5/3DES 2=Bcrypt/AES]) is 0 for all loaded hashes
Cost 2 (iteration count) is 1 for all loaded hashes

I wonder what happens to other JtR formats that use DES from OpenSSL, when run with latest OpenSSL. @DontBreakAlex On a system with OpenSSL 3.0.0+, can you please run ./john -test=0 -format=cpu to completion, and let us know which formats fail and how, as well as the final status line? Thank you!

[magnumripper]

If things were deprecated (to the point of not working) it should end up in build failure, no?

[magnumripper]

This bumps #2356.

In OpenCL, we have our own (nicked) algos for everything. The only reason that stopped us from doing so on CPU side is that some things (as noted in that issue) get a lot slower on certain platforms when using boiler plate code. This doesn't apply to SIMD platforms though, so is fairly uncommon.

I think I can actually find the old branch I experimented with mentioned in https://github.com/openwall/john/issues/2356#issuecomment-368418900 (or simply rewrite it). We could go from there.

[DontBreakAlex]

@solardiz Here are the tests you asked for:

➜  run git:(bleeding-jumbo) ✗ ./john -test=0 -format=cpu > test_output
Will run 32 OpenMP threads
Warning: cryptoSafe format should always be UTF-8. Use --target-encoding=utf8
Warning: sapg format should always be UTF-8. Use --target-encoding=utf8
Warning: saph format should always be UTF-8. Use --target-encoding=utf8
➜  run git:(bleeding-jumbo) ✗ grep FAIL test_output 
Testing: dmg, Apple DMG [PBKDF2-SHA1 256/256 AVX2 8x 3DES/AES]... (32xOMP) FAILED (cmp_all(1))
Testing: DPAPImk, DPAPI masterkey file v1 and v2 [SHA1/MD4 PBKDF2-(SHA1/SHA512)-DPAPI-variant 3DES/AES256 256/256 AVX2 8x]... (32xOMP) FAILED (cmp_all(1))
Testing: gpg, OpenPGP / GnuPG Secret Key [32/64]... (32xOMP) FAILED (cmp_all(11))
Testing: keychain, Mac OS X Keychain [PBKDF2-SHA1 3DES 256/256 AVX2 8x]... (32xOMP) FAILED (cmp_all(1))
Testing: Mozilla, Mozilla key3.db [SHA1 3DES 32/64]... (32xOMP) FAILED (cmp_all(1))
Testing: MSCHAPv2, C/R [MD4 DES (ESS MD5) 256/256 AVX2 8x3]... FAILED (valid (before init))
Testing: nethalflm, HalfLM C/R [DES 32/64]... (32xOMP) FAILED (cmp_all(1))
Testing: netlm, LM C/R [DES 32/64]... (32xOMP) FAILED (cmp_all(1))
Testing: netntlm, NTLMv1 C/R [MD4 DES (ESS MD5) 256/256 AVX2 8x3]... FAILED (valid (before init))
Testing: o10glogon, Oracle 10g-logon protocol [DES-AES128-MD5 32/64]... (32xOMP) FAILED (cmp_all(1))
Testing: o3logon, Oracle O3LOGON protocol [SHA1 DES 32/64]... (32xOMP) FAILED (cmp_all(1))
Testing: oracle, Oracle 10 [DES 32/64]... (32xOMP) FAILED (cmp_all(1))
Testing: PEM, PKCS#8 private key (RSA/DSA/ECDSA) [PBKDF2-SHA1 256/256 AVX2 8x 3DES/AES]... (32xOMP) FAILED (cmp_all(1))
Testing: sappse, SAP PSE [PKCS#12 PBE (SHA1) 256/256 AVX2 8x 3DES]... (32xOMP) FAILED (cmp_all(1))
Testing: SSH, SSH private key [RSA/DSA/EC/OPENSSH 32/64]... (32xOMP) FAILED (cmp_all(2))
15 out of 416 tests have FAILED

[magnumripper]

https://wiki.openssl.org/index.php/OpenSSL_3.0#Low_Level_APIs Use of the low level APIs has been informally discouraged by the OpenSSL development team for a long time. However in OpenSSL 3.0 this is made more formal. All such low level APIs have been deprecated. You may still use them in your applications, but you may start to see deprecation warnings during compilation (dependent on compiler support for this). Deprecated APIs may be removed from future versions of OpenSSL so you are strongly encouraged to update your code to use the high level APIs instead.

https://wiki.openssl.org/index.php/OpenSSL_3.0#Legacy_Algorithms Some cryptographic algorithms that were available via the EVP APIs are now considered legacy and their use is strongly discouraged. These legacy EVP algorithms are still available in OpenSSL 3.0 but not by default. If you want to use them then you must load the legacy provider. This can be as simple as a config file change, or can be done programmatically (see below).

[magnumripper]

@DontBreakAlex didn't you get a load of deprecation warnings (or other warnings) from OpenSSL when building JtR?

[solardiz]

@DontBreakAlex According to OpenSSL developers (see that OpenSSL issue I opened), things are supposed to work (you should only be getting deprecation warnings at build time, and that's it). Trying to investigate this, I see that the 15 formats failing for you (per your comment above) are not all formats where we use OpenSSL's DES. There are a few more: as400-des, RACF, RVARY, VNC, krb4, krb5 (several krb5 formats, some use DES some maybe not). Can you please let us know whether these pass tests for you (or maybe were somehow not tested)? You might want to simply attach the full test_output. Thank you!

[solardiz]

@DontBreakAlex Separately, can you please also repeat the test (of all formats) with OMP_NUM_THREADS=1? We need both output files (single-threaded and multi-thread). Thank you!

[solardiz]

@DontBreakAlex We also need the full build log, please. Especially all of the warnings. Thanks.

[DontBreakAlex]

@magnumripper I got no warnings while building, maybe this is because of the flag -DCL_SILENCE_DEPRECATION ? I'll post full build logs.

[magnumripper]

maybe this is because of the flag -DCL_SILENCE_DEPRECATION ?

I don't think so, it should only silence warnings from OpenCL on macOS.

[solardiz]

I got no warnings while building, maybe this is because of the flag -DCL_SILENCE_DEPRECATION?

No. Then my best guess is you're building against headers from older OpenSSL, not OpenSSL 3. However, this probably shouldn't result in the kind of runtime misbehavior you're seeing anyway. So my request for those tests and logs holds. When you're done with that, please also repeat for consistent OpenSSL headers+library (that is, with OpenSSL 3 headers). Thanks.

[solardiz]

@magnumripper If you have time, maybe try to reproduce the issue on your own? Unfortunately, I don't currently have enough time for this.

[magnumripper]

I might have some time tomorrow, we'll see

[DontBreakAlex]

@solardiz Here are the logs you requested (I captured both stdout and stderr): test_output.txt test_output_OMP_1.txt make_logs.txt configure_logs.txt

[magnumripper]

Then my best guess is you're building against headers from older OpenSSL, not OpenSSL 3

The below snippets from OP and https://github.com/openwall/john/issues/4831#issuecomment-927184594 are what we got from the headers, so they didn't mismatch. If they would, the lines would have been amended with somthing like printf("\t(loaded: %09lx)", OpenSSL_version_num()); and printf("\t(loaded: %s)", OpenSSL_version(OPENSSL_VERSION)); respectively.

./john --list=build-info
(...)
OpenSSL library version: 030000000
OpenSSL 3.0.0-dev xx XXX xxxx

./john --list=build-info
(...)
OpenSSL library version: 030000000
OpenSSL 3.0.0 7 sep 2021

[DontBreakAlex]

@solardiz I don't know autotools much, how do I "repeat for consistent OpenSSL headers+library (that is, with OpenSSL 3 headers)" ? Can I specify headers through a env variable ?

[magnumripper]

@solardiz I don't know autotools much, how do I "repeat for consistent OpenSSL headers+library (that is, with OpenSSL 3 headers)" ? Can I specify headers through a env variable ?

I think what he meant was that if you had built with older headers against newer OpenSSL, you should rebuild with them in sync. But like I said in previous post, that doesn't seem to be the case.

[magnumripper]

OK I can reproduce the problem exactly. I built with Homebrew's openssl@3 without a single build warning, and got this:

build-info:

OpenSSL library version: 030000000
OpenSSL 3.0.0 7 sep 2021

tests:

$ ../run/john -test=0 -form:cpu,-dynamic | grep -v PASS
Will run 16 OpenMP threads
Testing: dmg, Apple DMG [PBKDF2-SHA1 256/256 AVX2 8x 3DES/AES]... (16xOMP) FAILED (cmp_all(1))
Testing: DPAPImk, DPAPI masterkey file v1 and v2 [SHA1/MD4 PBKDF2-(SHA1/SHA512)-DPAPI-variant 3DES/AES256 256/256 AVX2 8x]... (16xOMP) FAILED (cmp_all(1))
Testing: gpg, OpenPGP / GnuPG Secret Key [32/64]... (16xOMP) FAILED (cmp_all(13))
Testing: keychain, Mac OS X Keychain [PBKDF2-SHA1 3DES 256/256 AVX2 8x]... (16xOMP) FAILED (cmp_all(1))
Testing: Mozilla, Mozilla key3.db [SHA1 3DES 32/64]... (16xOMP) FAILED (cmp_all(1))
Testing: MSCHAPv2, C/R [MD4 DES (ESS MD5) 256/256 AVX2 8x3]... FAILED (valid (before init))
Testing: nethalflm, HalfLM C/R [DES 32/64]... (16xOMP) FAILED (cmp_all(1))
Testing: netlm, LM C/R [DES 32/64]... (16xOMP) FAILED (cmp_all(1))
Testing: netntlm, NTLMv1 C/R [MD4 DES (ESS MD5) 256/256 AVX2 8x3]... FAILED (valid (before init))
Testing: o10glogon, Oracle 10g-logon protocol [DES-AES128-MD5 32/64]... (16xOMP) FAILED (cmp_all(1))
Testing: o3logon, Oracle O3LOGON protocol [SHA1 DES 32/64]... (16xOMP) FAILED (cmp_all(1))
Testing: oracle, Oracle 10 [DES 32/64]... (16xOMP) FAILED (cmp_all(1))
Testing: PEM, PKCS#8 private key (RSA/DSA/ECDSA) [PBKDF2-SHA1 256/256 AVX2 8x 3DES/AES]... (16xOMP) FAILED (cmp_all(1))
Testing: sappse, SAP PSE [PKCS#12 PBE (SHA1) 256/256 AVX2 8x 3DES]... (16xOMP) FAILED (cmp_all(1))
Testing: SSH, SSH private key [RSA/DSA/EC/OPENSSH 32/64]... (16xOMP) FAILED (cmp_all(2))
15 out of 257 tests have FAILED

[solardiz]

@DontBreakAlex I agree with @magnumripper - you probably already do have those consistent, so this cancels my request to rebuild differently.

So you're not getting any deprecation warnings. Also, there's no difference in failing formats by number of OpenMP threads (1 vs. 32) - it's the 15 formats anyway. And those other DES-using formats I mentioned pass tests. That's puzzling.

@magnumripper Great! I think try -fno-strict-aliasing and see if makes a difference.

[magnumripper]

@magnumripper Great! I think try -fno-strict-aliasing and see if makes a difference.

It didn't. Still no warnings, still same failures.

[magnumripper]

I don't have much more time now but I did try memsetting a buffer to 0 before calling 3des where the DMG format fails, and there were still non-zero output - although it's somehow not the correct output.

[claudioandre-br]

We're actually hiding depreciation notices. A lot of them.

• From configure:

  checking if gcc supports --param allow-store-data-races=0... no
  checking if gcc supports -Wno-deprecated-declarations... yes  <=====
  checking if gcc supports -Wformat-extra-args... yes

Disable it, then:

[solardiz]

Here's a new guess: maybe it isn't that some of our uses of OpenSSL DES fail and some work right, but rather that some inputs to DES produce the correct behavior and some don't. So the difference is in our test vectors for the different formats triggering the problematic inputs to the OpenSSL DES calls or not. If so, maybe running with additional inputs (more than our test vectors) will uncover failures even for some formats that currently pass tests.

[magnumripper]

Good catch @claudioandre-br, that explains why we don't get the warnings. Still, it shouldn't fail like it does.

[magnumripper]

This after removing `-Wno-deprecated-declarations´ ossl3-build-warnings.txt

[magnumripper]

Our use of -Wno-deprecated-declarations seems to stem all the way back to ecdadb503d2 in 2013

[magnumripper]

Our use of -Wno-deprecated-declarations seems to stem all the way back to ecdadb5 in 2013

No, it's even older. It's from 11b0e6409 in 2012

[claudioandre-br]

To make SSH pass self-test, I need to disable tests starting with $sshng$0$8$.

[claudioandre-br]

I would say [1] that we can prove to the openssl guys that (at least) DES_set_key() is behaving differently in 3.0.

A printf() at line 325 seems ok when using 1.1 and strange in 3.0.

[1] Well, almost everything looks random in this encryption mechanism. Odd, not sure I got it right. Anyway, I guess a simple test case can prove SSL lib is misbehaving using the first $sshng$0$8$ hash (key is television).

[solardiz]

I wonder if this has anything to do with DES key parity. Regardless, we need to come up with a reduced test case - a separate program that shows different behavior on OpenSSL 1.x vs. 3.0.

[magnumripper]

I ripped some code and test vector data from mozilla format and ended up with this (cross posted to https://github.com/openssl/openssl/issues/16859)

#include <openssl/des.h>
#include <string.h>
#include <stdio.h>

int main(int argc, char **argv)
{
    DES_cblock ivec;
    DES_key_schedule ks1, ks2, ks3;
    unsigned int key[10] = {
        0x09696814, 0x434ab064, 0x562a405d, 0xc468793e, 0x3131685a,
        0x1a1f3239, 0x6d84bd6e, 0xc69c682d, 0x89e2684d, 0xc8f98512
    };
    unsigned int out[4];
    unsigned int correct[4] = {
        0xbddeeb9d, 0x78b29645, 0x2b9b02de, 0x2ece8522
    };

    DES_set_key((DES_cblock *) key, &ks1);
    DES_set_key((DES_cblock *) &key[2], &ks2);
    DES_set_key((DES_cblock *) &key[4], &ks3);
    memcpy(ivec, &key[8], 8);

    DES_ede3_cbc_encrypt((unsigned char*)"password-check\x02\x02", (unsigned char*)out, 16,
                         &ks1, &ks2, &ks3, &ivec, DES_ENCRYPT);

    int result = memcmp(out, correct, 16);

    fprintf(stderr, "Out: %08x %08x %08x %08x (%s)\n", out[0], out[1], out[2], out[3],
            result ? "wrong" : "correct");

    return result;
}

Here's the correct result from older OpenSSL:

$ gcc -lcrypto -I/usr/local/opt/openssl@1.1/include -L/usr/local/opt/openssl@1.1/lib -o des-test des-test.c && ./des-test 
Out: bddeeb9d 78b29645 2b9b02de 2ece8522 (correct)

OpenSSL 3 doesn't only give a different result, but a different one each time:

$ gcc -lcrypto -I/usr/local/opt/openssl@3/include -L/usr/local/opt/openssl@3/lib -Wno-deprecated-declarations -o des-test des-test.c && ./des-test 
Out: 51f2626e 2b1dfae9 bc27db32 5c108382 (wrong)
$ ./des-test 
Out: d36c8778 a62c87b5 42439abd 673d3334 (wrong)
$ ./des-test 
Out: c8c63db1 50674fe6 e1174e80 2212a6af (wrong)
$ ./des-test 
Out: 6fbec757 443aff88 555dd1cf 243a5a5f (wrong)
$ ./des-test 
Out: 1c65b31b b72d3d7c df43e6ec 528f2387 (wrong)

[solardiz]

Thanks, @magnumripper! That example uses 3DES, but some of our failing formats use plain DES. So perhaps a smaller test case is possible?

[magnumripper]

They already found the problem using this, and I think it also explains why some formats didn't fail.

[magnumripper]

Confirmed:

$ ../run/john -test=0 -form:@des,-opencl | grep -v PASS
Testing: dmg, Apple DMG [PBKDF2-SHA1 3DES/AES 32/64]... FAILED (cmp_all(1))
Testing: DPAPImk, DPAPI masterkey file v1 and v2 [SHA1/MD4 PBKDF2-(SHA1/SHA512)-DPAPI-variant 3DES/AES256 32/64]... FAILED (cmp_all(1))
Testing: keychain, Mac OS X Keychain [PBKDF2-SHA1 3DES 32/64]... FAILED (cmp_all(1))
Testing: Mozilla, Mozilla key3.db [SHA1 3DES 32/64]... FAILED (cmp_all(1))
Testing: MSCHAPv2, C/R [MD4 DES (ESS MD5) 32/64]... FAILED (valid (before init))
Testing: nethalflm, HalfLM C/R [DES 32/64]... FAILED (cmp_all(1))
Testing: netlm, LM C/R [DES 32/64]... FAILED (cmp_all(1))
Testing: netntlm, NTLMv1 C/R [MD4 DES (ESS MD5) 32/64]... FAILED (valid (before init))
Testing: o10glogon, Oracle 10g-logon protocol [DES-AES128-MD5 32/64]... FAILED (cmp_all(1))
Testing: o3logon, Oracle O3LOGON protocol [SHA1 DES 32/64]... FAILED (cmp_all(1))
Testing: oracle, Oracle 10 [DES 32/64]... FAILED (cmp_all(1))
Testing: PEM, PKCS#8 private key (RSA/DSA/ECDSA) [PBKDF2-SHA1 32/64 3DES/AES]... FAILED (cmp_all(1))
Testing: sappse, SAP PSE [PKCS#12 PBE (SHA1) 32/64 3DES]... FAILED (cmp_all(1))
13 out of 31 tests have FAILED

$ git grep -lF 'DES_set_key(' | xargs sed -ri 's/DES_set_key/DES_set_key_unchecked/g'
$ make -sj8

Make process completed.
$ ../run/john -test=0 -form:@des,-opencl | grep -v PASS
All 31 formats passed self-tests!

[magnumripper]

13 out of 31 tests have FAILED

This makes me wonder if some of the formats that did NOT fail also could do something like:

if (DES_set_key_checked(key) < 0) /* -1 is wrong parity, -2 is weak key */
    goto reject;

For some formats it might be a boost, if we can skip trying decryption of "a lot" of data (I didn't check if this really applies to any of those formats). But before doing anything like that we'd need to be damn sure about it so we don't introduce false negatives just not caught by whatever test vectors we happen to have.

Here is the list of the formats that did not have problems:

$ ../run/john -test=0 -form:@des,-opencl | grep -v FAIL
Testing: descrypt, traditional crypt(3) [DES 64/64]... PASS
Testing: bsdicrypt, BSDI crypt(3) ("_J9..", 725 iterations) [DES 64/64]... PASS
Testing: LM [DES 64/64]... PASS
Testing: AFS, Kerberos AFS [DES 48/64 4K]... PASS
Testing: tripcode [DES 64/64]... PASS
Testing: as400-des, AS/400 DES [DES 32/64]... PASS
Testing: krb4, Kerberos v4 TGT [DES 32/64]... PASS
Testing: krb5, Kerberos v5 TGT [3DES 32/64]... PASS
Testing: krb5-17, Kerberos 5 DB etype 17 [DES / PBKDF2-SHA1 32/64 AES]... PASS
Testing: krb5-18, Kerberos 5 DB etype 18 [DES / PBKDF2-SHA1 32/64 AES]... PASS
Testing: krb5-3, Kerberos 5 DB etype 3 [DES / PBKDF2-SHA1 32/64 AES]... PASS
Testing: mdc2, MDC-2 [MDC-2DES]... PASS
Testing: mschapv2-naive, MSCHAPv2 C/R [MD4 DES 64/64 naive]... PASS
Testing: netntlm-naive, NTLMv1 C/R [MD4 DES (ESS MD5) DES 64/64 naive]... PASS
Testing: RACF [DES 32/64]... PASS
Testing: RACF-KDFAES [KDFAES (DES + HMAC-SHA256/64 + AES-256)]... PASS
Testing: RVARY [DES 32/64]... PASS
Testing: VNC [DES 32/64]... PASS

Also, note to self: The --format=@des used in these tests didn't catch the GPG and SSH formats (which both fail) so for further tests better use --format:gpg,ssh,@des,+cpu.

[solardiz]

BTW, I did write "I wonder if this has anything to do with DES key parity." in a comment here 7 days ago. I should have probably mentioned that in the OpenSSL issue as well - could have brought us to the present understanding sooner.

This makes me wonder if some of the formats that did NOT fail also could do something like:

if (DES_set_key_checked(key) < 0) /* -1 is wrong parity, -2 is weak key */
    goto reject;

Maybe for some, but I doubt it. Generally, it's expected that wrong parity DES keys would be silently repaired (or parity bits ignored, as the actual DES algorithm doesn't use them anyway), and weak ones used anyway - not only in JtR, but also in the original implementations.

[magnumripper]

Maybe for some, but I doubt it

I really doubt it too. If we mimiced the original KDF, that's the key it too would end up with. And anyway, none of the not failing formats are FMT_HUGE.