openssl-enc: Support recent OpenSSL's encrypted files, or at least refuse to work on them

[claudioandre-br]

Well, # 4414 shows us that a commit with untested code was merged. So, I:

$ openssl aes-256-cbc -a -salt -in 4414.patch -out secrets.txt.enc

And I noticed that in addition to dozens of false positives per second, the correct password was not found.

$ john teste --mask=openw?a?ll
Note: This format may emit false positives, so it will keep trying even after finding a
possible candidate.
Using default input encoding: UTF-8
Loaded 1 password hash (openssl-enc, OpenSSL "enc" encryption [32/64])
Warning: poor OpenMP scalability for this hash type, consider --fork=8
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
openw%nl         (secrets.txt.enc)
openw+rl         (secrets.txt.enc)
openw~sl         (secrets.txt.enc)
openw)ml         (secrets.txt.enc)
openwrbl         (secrets.txt.enc)
openwUvl         (secrets.txt.enc)
openw6fl         (secrets.txt.enc)
openwzxl         (secrets.txt.enc)
8g 0:00:00:00 N/A 25.00g/s 7718p/s 7718c/s 7718C/s openw!fl..openw|ql
Session completed. 

[solardiz]

Well, #4414 shows us that a commit with untested code was merged

That was 4ef1ef28f91034cebce396f81456c540f00355ab, which modified many 2john tools at once. We need to (re)review this commit now. Edit: see also #4200.

[solardiz]

I cannot reproduce the problem in Claudio's first comment here - I am getting the correct password cracked:

$ openssl aes-256-cbc -a -salt -in fuzz.dic -out secrets.txt.enc
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
[solar@super run]$ ./openssl2john.py secrets.txt.enc 
secrets.txt.enc:$openssl$0$0$8$cb0edd45ad353541$18f6794590e63fe647f3657433304d12762284251e72f1ecb8fdd1318a2b4cc4$0$256$542ba121e464f7e5b71e387fafb47cce03d4b837e66964f3897bed80be13b050482b31b1bd6b698c441a986f1f6316e6b3bae1ae7589303fd887932c392fb1216b14a2ebd4d1d176384419a1725f3a338bd6cb66b0b0cf3aec769bc6dea8de4357266eaa7ba4454b213dae3426733133a908fa553e42f20b852389b30f0d691df37e54fc8c949083ec8f1c392c73f365a14c96d83fa0f2406c9abb5921bceed95d9fad61d4b6aa33a74efc955ce402d610dd672e9077982727a35916dbfbfe21a888d46a26f4962da5010b6128e3ef2bef9e348514c510682114f199f81fff67d0e2c2cf084cbf308a5ea3a7030898847da7c1e5b8f0970d8b8a0ad919500851$0
[solar@super run]$ ./openssl2john.py secrets.txt.enc > teste
[solar@super run]$ ./john teste --mask=openw?a?ll
Note: This format may emit false positives, so it will keep trying even after finding a possible candidate.
Using default input encoding: UTF-8
Loaded 1 password hash (openssl-enc, OpenSSL "enc" encryption [32/64])
Warning: poor OpenMP scalability for this hash type, consider --fork=32
Will run 32 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
openwSrl         (secrets.txt.enc)     
openwall         (secrets.txt.enc)     
openw]tl         (secrets.txt.enc)     
openw:dl         (secrets.txt.enc)     
openw,yl         (secrets.txt.enc)     
openw]bl         (secrets.txt.enc)     
openw~xl         (secrets.txt.enc)     
openw`ql         (secrets.txt.enc)     
8g 0:00:00:00  38.09g/s 11761p/s 11761c/s 11761C/s openwaal..openw|ql
Session completed. 

As you can see, "openwall" appears among the 8 possible guesses. The other 7 are false positives.

The false positives are gone with:

[solar@super run]$ ./openssl2john.py -a 90 secrets.txt.enc > teste
[solar@super run]$ ./john teste --mask=openw?a?ll
Note: This format may emit false positives, so it will keep trying even after finding a possible candidate.
Using default input encoding: UTF-8
Loaded 1 password hash (openssl-enc, OpenSSL "enc" encryption [32/64])
Warning: poor OpenMP scalability for this hash type, consider --fork=32
Will run 32 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
openwall         (secrets.txt.enc)     
1g 0:00:00:00  4.761g/s 11761p/s 11761c/s 11761C/s openwaal..openw|ql
Session completed. 

Claudio, can you try reproducing this issue again - somehow the issue did occur for you? Is it maybe triggered by the specific input file you had?

[claudioandre-br]

Running it right now:

$ wget https://github.com/openwall/john/pull/4414.patch
$ openssl aes-256-cbc -a -salt -in 4414.patch -out secrets.txt.enc
$ /snap/john-the-ripper/current/run/openssl2john.py secrets.txt.enc > teste
$ john teste --mask=openw?a?ll
Note: This format may emit false positives, so it will keep trying even after finding a possible candidate.
Using default input encoding: UTF-8
Loaded 1 password hash (openssl-enc, OpenSSL "enc" encryption [32/64])
Warning: poor OpenMP scalability for this hash type, consider --fork=8
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
openwiul         (secrets.txt.enc)     
openwIul         (secrets.txt.enc)     
openw7bl         (secrets.txt.enc)     
openwnjl         (secrets.txt.enc)     
openw1fl         (secrets.txt.enc)     
openwCwl         (secrets.txt.enc)     
6g 0:00:00:00 N/A 11.76g/s 4843p/s 4843c/s 4843C/s openw!fl..openw|ql
Session completed. 

$ python --version
Python 2.7.18

$ john | head -n1
John the Ripper 1.9.0-jumbo-1+bleeding-e93ee0c 2021-04-23 12:05:23 +0200 OMP [linux-gnu 64-bit x86_64 AVX2 AC]

python3 /snap/john-the-ripper/current/run/openssl2john.py secrets.txt.enc
secrets.txt.enc:$openssl$0$0$8$3db72351df472df0$e2a4f8e247c3312fe461021fc7c51e1a60c731c99862db6051f340d22837d8d2$0$256$6e8b4b00f97e576b1496bf55f4d64dbe6f74c99f0e2af096fff044a17ab7153408fac30cd460576e1c2871144f3284840a663e606c213babb5c3cac8cf7934c2b50f95c4693327cfdd1d580c3344225d7345f47c498b3d8c2ed59fd0ca8b7f814331c958b994eabd1ea2cd9ba6675ebbfd4c8ef658139539df05774317b3aa7782cb2bd8e0092f94ce757ae9bbaa13d0bd667359f078031c5bb94445abc60ff00095ce5adcd40b9bddcc72f01b78620a0a73bce53270f748bf2ded909036d32d78cfb8bf04e27eb5a8bbcb294b7f1962c0512b9a2e108572ad8b3f04ef15e279ae13c716400e4f6649b14fcc4342cd0a22c834de9aa4f370edd4f874c9d78de7$0

The secrets.txt.enc file: secrets.txt.zip

[claudioandre-br]

I just tested the file using a openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new

[solardiz]

I just tested the file using a openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new

Are you able to decrypt your file using password "openwall"? I downloaded your encrypted file from here, and I can't decrypt it with that password:

[solar@super run]$ openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new
enter aes-256-cbc decryption password:
bad decrypt
139890611070792:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:596:

[claudioandre-br]

I can, as you can see below:

$ rm -f secrets.txt.new; openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new -k openwall; cat secrets.txt.new
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
From 1affca4a0d8329e1910d54c1e13c2b27de3b3db8 Mon Sep 17 00:00:00 2001
From: Elliot Mason <ekid113@yahoo.com>
Date: Sun, 25 Oct 2020 00:53:01 -0500
Subject: [PATCH] python string formatting

---
 run/openssl2john.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/run/openssl2john.py b/run/openssl2john.py
index 7e754c0910..15406df29d 100755
--- a/run/openssl2john.py
+++ b/run/openssl2john.py
@@ -78,7 +78,7 @@ def process(filename, plaintext=None, cipher=0, md=0, minascii=0):
             if PY3:
                 rdata = rdata.decode("ascii")
             sys.stdout.write("%s:$openssl$%s$%s$8$%s$%s$0$%s$%s$%s\n" %
-                             (os.path.basename(filename,) cipher, md, salt,
+                             (os.path.basename(filename), cipher, md, salt,
                               last_chunk,
                               len(rdata) // 2, rdata, s))

[solardiz]

Tried downloading to and decrypting on another system (different OpenSSL), still can't. Can others reading this please try this as well, and comment in here?

[magnumripper]

I also can't

$ openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new
WARNING: can't open config file: /usr/local/openssl10/openssl.cnf
enter aes-256-cbc decryption password:
bad decrypt
4474826412:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:467:

$ openssl version -a
WARNING: can't open config file: /usr/local/openssl10/openssl.cnf
OpenSSL 1.0.0m 5 Jun 2014
built on: Wed Sep 30 19:59:33 CEST 2020
platform: darwin64-x86_64-cc
options:  bn(64,64) rc4(ptr,char) des(idx,cisc,16,int) idea(int) blowfish(idx) 
compiler: cc -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -arch x86_64 -O3 -DL_ENDIAN -DMD32_REG_T=int -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DWHIRLPOOL_ASM
OPENSSLDIR: "/usr/local/openssl10"

[claudioandre-br]

Below, I tested using Docker Playground (aka, play with Docker). Both worked. Later, I will try using AWS ami.

BTW: Fedora is latest (right now it is F34).

docker run -it ubuntu:20.10
apt update ; apt install -y wget unzip
wget https://github.com/openwall/john/files/6452914/secrets.txt.zip

unzip secrets.txt.zip
rm -f secrets.txt.new; openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new -k openwall; cat secrets.txt.new

docker run -it fedora
dnf install -y wget unzip openssl
wget https://github.com/openwall/john/files/6452914/secrets.txt.zip

unzip secrets.txt.zip
rm -f secrets.txt.new; openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new -k openwall; cat secrets.txt.new

[claudioandre-br]

The AWS ami fails. It seems that old versions of openssl behave differently. I use 2020+ versions.

AMIs version:

openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

[frank-dittrich]

Fedora F32 also has a recent openssl version.

$ openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new
enter aes-256-cbc decryption password:
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
$ ls -l secrets.txt.*
-rw-rw-r--. 1 fd fd 1260 10. Mai 16:37 secrets.txt.enc
-rw-rw-r--. 1 fd fd  898 10. Mai 18:35 secrets.txt.new
-rw-rw-r--. 1 fd fd 1191 10. Mai 18:34 secrets.txt.zip
[fd@localhost run]$ $ openssl version -a
OpenSSL 1.1.1k  FIPS 25 Mar 2021
built on: Fri Mar 26 00:00:00 2021 UTC
platform: linux-x86_64
options:  bn(64,64) md2(char) rc4(16x,int) des(int) idea(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wa,--noexecstack -Wa,--generate-missing-build-notes=yes -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" -DSYSTEM_CIPHERS_FILE="/etc/crypto-policies/back-ends/openssl.config"
OPENSSLDIR: "/etc/pki/tls"
ENGINESDIR: "/usr/lib64/engines-1.1"
Seeding source: os-specific
engines:  rdrand dynamic 

[solardiz]

@frank-dittrich You need to take a look at the decrypted file - just because openssl agreed to create it doesn't mean it decrypted correctly. For our false positives, it also agrees to decrypt, but the resulting data is obviously incorrect. That said, given that I was getting a decryption error (meaning the padding was wrong for me), we already know that your decryption is at least different from mine, so is probably correct.

Also this is interesting:

*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.

Perhaps we need to add support for newer key derivation? (Not used in this file, but supported by openssl enc in newer versions like yours.)

[frank-dittrich]

$ file secrets.txt.new 
secrets.txt.new: unified diff output, ASCII text
$ sha1sum secrets.txt.new 
fcf416cb513c650b4e50d7b734d536bb1b1cb8e4  secrets.txt.new

The file contents is identical to the one listed by @claudioandre-br in https://github.com/openwall/john/issues/4415#issuecomment-836828532

[frank-dittrich]

$ openssl enc -help 2>&1|grep -- -pbkdf2
 -pbkdf2             Use password-based key derivation function 2
$ openssl enc -help 2>&1|grep -- -iter
 -iter +int          Specify the iteration count and force use of PBKDF2

From the man page:

       -iter count
           Use a given number of iterations on the password in deriving the encryption key.  High values increase the time required to brute-force the resulting file.  This option enables the use of PBKDF2 algorithm to derive
           the key.

       -pbkdf2
           Use PBKDF2 algorithm with default iteration count unless otherwise specified.

Do you want me to create a new pbkdf2 sample file? Let me know the exact command (incl. iteration count) you want me to use.

[solardiz]

Apparently, the default -md algorithm changed from MD5 to SHA-256. I am able to decrypt Claudio's file by specifying -md sha256 to openssl (old version):

[solar@super run]$ openssl aes-256-cbc -d -a -in secrets.txt.enc -out secrets.txt.new -k openwall -md sha256
[solar@super run]$ file secrets.txt.new
secrets.txt.new: unified diff output text

I am also able to crack it with JtR by specifying -m 2 to openssl2john.py:

[solar@super run]$ ./openssl2john.py -m 2 -a 90 secrets.txt.enc > teste
[solar@super run]$ ./john teste --mask=openw?a?ll
Note: This format may emit false positives, so it will keep trying even after finding a possible candidate.
Using default input encoding: UTF-8
Loaded 1 password hash (openssl-enc, OpenSSL "enc" encryption [32/64])
Warning: poor OpenMP scalability for this hash type, consider --fork=32
Will run 32 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
openwall         (secrets.txt.enc)     
1g 0:00:00:00  6.666g/s 16466p/s 16466c/s 16466C/s openwaal..openw|ql
Session completed. 

Apparently, there's inherently no reliable way to tell which MD algorithm and which cipher and mode(?) were used for encryption until we've found the correct password (and in some cases not even when we did).

So maybe we should have both openssl2john.py and john print warnings explaining this and the need to use the right options to openssl2john.py.

I haven't yet looked into the PBKDF2 stuff. Is its use and its iteration count somehow encoded in the file, or does the user have to manually supply the correct options to achieve correct decryption? Frank, can you please try encrypting a file using those options and then see if you're able to decrypt it without giving the same options? Thanks!

I think we'll need to update this issue's title again, but I'm not yet sure what it should be.