pem2john and PEM formats blindly assume SHA-1 instead of encoding/using actual "prf algorithm" (can be SHA-256)

[exploide]

While working on #4833 I encountered a problem with cracking openssl encrypted private key files. It does work with the example key but not with a newly generated test key. The password is "test".

> openssl req -newkey rsa:2048 -keyout keynamehere.key -out csrnamehere.key
Generating a RSA private key
................................+++++
..........................................+++++
writing new private key to 'keynamehere.key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

> cat keynamehere.key 
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFHDBOBgkqhkiG9w0BBQ0wQTApBgkqhkiG9w0BBQwwHAQIDow6HN28d5ACAggA
MAwGCCqGSIb3DQIJBQAwFAYIKoZIhvcNAwcECEwJ2k3MnATOBIIEyE04z3UvKMus
5bRcTpU6CWgPIbAhyclZpvr6oI62hr6sMkMqQaq3q3Y1JN5W8C64PLJkS7yTAKIf
k1NT/mFFvFn2i+V+SYgF7TUHERJugZeLwl3BGsvBLpgvW5QKr3vuCEWNAlcFFBH9
7hoKXscuyOIIh0yrceCH3/2SMT1eHy/WdxARrEuX+5oPkQYFp0DDI/EnMtB7MBPh
6Ss7R+Ho73EUI+Zw/yxEQmtbFLdOrALe8DhFO5RD9zZoV7q311MnJuHb/ISrF5Rg
srKXIZyx4cB+kHV5d881sAw/4DwA+1k5ZdMRWoZ5Ha7iYGOmjOlgKO9lodPeFRuE
aVBFViCUQ5SDeBbVzJ1e8xyaui247ylAmX/HTI/hlxtmUQVj80x0vC5zQn5MGtUR
0/lH8SYnIHQLWNtnJDo0R3nxIqe4gDdHsiQpBEJ0rjSgLtNhLz5qoAL2SQanwrjr
GZhA3XsRuNUz8c5xd5qN0HmF5kN5fSQqbrUge0pGuudG6bs+/wVdGhwlg4X4mBlJ
U5jUGftZiqvpPqtciG1lom/TdXQ5b487fNQFCAwtGAPHKjJSmYp+MzvULpzvSqbr
z1DLUOxrrCJFD1PqCkfxxIsLwS0Gzx053CaPJpfYULbbjx5T1WZd0lk8M0eNsh3w
sc9T5wP+YQyC+PuI5nAwvcYy8UaWaQBDlLg71vzgzObOcpIJQxG8DgvcRRL6sz5L
gvQMYXo2E6UUQXxmz32r8bRWL0J7ObQ89Q8HRc+hW0ovYV8TS9uUU5qPFYi6bzoj
FtvSrViG1dw5uyzM3/1swp+hGg+k2LtkNa26IZb9S/D5s5p8DBH7ud2G+yFRUNJD
F+FuE3DyryB43DFVQw1gEjYPFUhQWG/5TUMCln6hBjEMREB7KcN40XZ+srXhgq6+
88O2gUH0PhHpkZTujO20OUQhU5mEmHcCdA68L9jsNQ2vqq73scRfrnFdHZroXqHG
SExIG254UpsoHsw568s0kuGxYNvAxe4VGMqu5RmeOZgaS4BodZbTec/tq1leh7Bk
ClFdg4mpLm+pKxroA+siOtJyCkYjPHJW1s5zn4GofOuuq69KKul8k3O0/8H4DJEW
+I/cbI9CZRJa2aLQTyPPaZPjPsmUCnlZP1P8iJ+K4VcwtbfSADv9cqDsFjq7NpJS
V5PnGTeLdhx2J6cVJc51sKdVZYwYE6JbPZcEBaU1mLPDxOFcmkrqDHnoL6cMsAwa
PRL9J/mfL6yhPRInObj9Q9OOTdMq5ERzUCOZPYn4m0SA4P8Pd7NgZ2Y57+jsylkF
+wCshTTK9Kw4X3qBS0PcGC2dM6MOZDesVoZBXCPjYUJKgORDJyoCFMC9XY7CaCNp
6wUdDX08cZTJP35PA8c47uDbHchHFSb54xeOTIhxb+CQhskdxXwtpkR5C9XBCHYY
0C0z7BLrylhH7Us2E6NCZJ+2DKX+LNJZxn144rzU61Zn4l4UNLvscF41V5O9Cxua
r/TX4hZOadeXt3KM/GdH5GqJuxRg4bLG7p3MXuP9MQV8Wr3K2gHVfeuEJv20FfPP
KDX/Eqzj8k/4eNeE3MS4djFvEp6uJqFCsVJetRAzsXoj8TaVSQUxqlA1cdqouO99
EDcLtzI/hMEhKaEnM6y7Ug==
-----END ENCRYPTED PRIVATE KEY-----

> ~/Downloads/repos/john/run/pem2john.py keynamehere.key | tee hash
$PEM$1$1$0e8c3a1cddbc7790$2048$4c09da4dcc9c04ce$1224$4d38cf752f28cbace5b45c4e953a09680f21b021c9c959a6fafaa08eb686beac32432a41aab7ab763524de56f02eb83cb2644bbc9300a21f935353fe6145bc59f68be57e498805ed350711126e81978bc25dc11acbc12e982f5b940aaf7bee08458d0257051411fdee1a0a5ec72ec8e208874cab71e087dffd92313d5e1f2fd6771011ac4b97fb9a0f910605a740c323f12732d07b3013e1e92b3b47e1e8ef711423e670ff2c44426b5b14b74eac02def038453b9443f7366857bab7d7532726e1dbfc84ab179460b2b297219cb1e1c07e90757977cf35b00c3fe03c00fb593965d3115a86791daee26063a68ce96028ef65a1d3de151b846950455620944394837816d5cc9d5ef31c9aba2db8ef2940997fc74c8fe1971b66510563f34c74bc2e73427e4c1ad511d3f947f1262720740b58db67243a344779f122a7b8803747b22429044274ae34a02ed3612f3e6aa002f64906a7c2b8eb199840dd7b11b8d533f1ce71779a8dd07985e643797d242a6eb5207b4a46bae746e9bb3eff055d1a1c258385f89819495398d419fb598aabe93eab5c886d65a26fd37574396f8f3b7cd405080c2d1803c72a3252998a7e333bd42e9cef4aa6ebcf50cb50ec6bac22450f53ea0a47f1c48b0bc12d06cf1d39dc268f2697d850b6db8f1e53d5665dd2593c33478db21df0b1cf53e703fe610c82f8fb88e67030bdc632f1469669004394b83bd6fce0cce6ce7292094311bc0e0bdc4512fab33e4b82f40c617a3613a514417c66cf7dabf1b4562f427b39b43cf50f0745cfa15b4a2f615f134bdb94539a8f1588ba6f3a2316dbd2ad5886d5dc39bb2cccdffd6cc29fa11a0fa4d8bb6435adba2196fd4bf0f9b39a7c0c11fbb9dd86fb215150d24317e16e1370f2af2078dc3155430d6012360f154850586ff94d4302967ea106310c44407b29c378d1767eb2b5e182aebef3c3b68141f43e11e99194ee8cedb4394421539984987702740ebc2fd8ec350dafaaaef7b1c45fae715d1d9ae85ea1c6484c481b6e78529b281ecc39ebcb3492e1b160dbc0c5ee1518caaee5199e39981a4b80687596d379cfedab595e87b0640a515d8389a92e6fa92b1ae803eb223ad2720a46233c7256d6ce739f81a87cebaeabaf4a2ae97c9373b4ffc1f80c9116f88fdc6c8f4265125ad9a2d04f23cf6993e33ec9940a79593f53fc889f8ae15730b5b7d2003bfd72a0ec163abb3692525793e719378b761c7627a71525ce75b0a755658c1813a25b3d970405a53598b3c3c4e15c9a4aea0c79e82fa70cb00c1a3d12fd27f99f2faca13d122739b8fd43d38e4dd32ae444735023993d89f89b4480e0ff0f77b360676639efe8ecca5905fb00ac8534caf4ac385f7a814b43dc182d9d33a30e6437ac5686415c23e361424a80e443272a0214c0bd5d8ec2682369eb051d0d7d3c7194c93f7e4f03c738eee0db1dc8471526f9e3178e4c88716fe09086c91dc57c2da644790bd5c1087618d02d33ec12ebca5847ed4b3613a342649fb60ca5fe2cd259c67d78e2bcd4eb5667e25e1434bbec705e355793bd0b1b9aaff4d7e2164e69d797b7728cfc6747e46a89bb1460e1b2c6ee9dcc5ee3fd31057c5abdcada01d57deb8426fdb415f3cf2835ff12ace3f24ff878d784dcc4b876316f129eae26a142b1525eb51033b17a23f13695490531aa503571daa8b8ef7d10370bb7323f84c12129a12733acbb52

> echo test > wordlist

> ~/Downloads/repos/john/run/john --wordlist=wordlist hash
Using default input encoding: UTF-8
Loaded 1 password hash (PEM, PKCS#8 private key (RSA/DSA/ECDSA) [PBKDF2-SHA1 256/256 AVX2 8x 3DES/AES])
Cost 1 (iteration count) is 4096 for all loaded hashes
Cost 2 (cipher [1=3DES 2/3/4=AES-128/192/256]) is 1 for all loaded hashes
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
Warning: Only 1 candidate left, minimum 64 needed for performance.
0g 0:00:00:00 DONE (2021-09-26 16:11) 0g/s 25.00p/s 25.00c/s 25.00C/s test
Session completed. 

> ~/Downloads/repos/john/run/john --show hash
0 password hashes cracked, 1 left

> ~/Downloads/repos/john/run/john --list=build-info
Version: 1.9.0-jumbo-1+bleeding-edf64e869 2021-09-25 12:13:58 -0300
Build: linux-gnu 64-bit x86_64 AVX2 AC OMP
SIMD: AVX2, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
CPU tests: AVX2
$JOHN is /home/jannik/Downloads/repos/john/run/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 32768
SINGLE_BUF_MAX: 4294967295
Effective limit: Max. KPC 32768
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 11.2.1
GNU libc version: 2.33 (loaded: 2.33)
Crypto library: OpenSSL
OpenSSL library version: 0101010cf
OpenSSL 1.1.1l  FIPS 24 Aug 2021
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
times(2) sysconf(_SC_CLK_TCK) is 100
Using times(2) for timers, resolution 10 ms
HR timer: clock_gettime(), latency 125 ns
Total physical host memory: 15647 MiB
Available physical host memory: 12368 MiB
Terminal locale string: en_US.UTF-8
Parsed terminal locale: UTF-8

[solardiz]

Thanks, @exploide. This is a nasty issue. I added print(kdf_params) into pem2john.py, and here's what I see for your key above:

OrderedDict([('salt', b'\x0e\x8c:\x1c\xdd\xbcw\x90'), ('iteration_count', 2048), ('key_length', None), ('prf', OrderedDict([('algorithm', 'sha256'), ('parameters', None)]))])

Notice the sha256 there. For a key generated with older OpenSSL and crackable by current JtR, there's sha1 in that place.

We need to add encoding of this field into pem2john.py output (perhaps upgrading it from $PEM$1 to $PEM$2) and its decoding and reporting as a tunable cost and (most importantly) its actual usage in JtR formats pem and pem-opencl. As an option, it's OK to mostly exclude pem-opencl from the initial PR, only having it refuse to load "hashes" that use other than SHA-1 in there - and fix that with a follow-up PR (possibly by a different person). In fact, it's OK for the initial PR to even be limited to introducing the new encoding and its support in the formats, while having both formats refuse to actually process other than SHA-1 - that in itself would already be an improvement and would fix the bug (we'd have explicit rejection rather than incorrect processing), leaving only the enhancement (actual support) to be worked on.

@exploide Would you like to work on any/all of this? :-)

[exploide]

Notice the sha256 there. For a key generated with older OpenSSL and crackable by current JtR, there's sha1 in that place.

Ok, I expected something like this. Good that it doesn't affect all PEM keys.

Would you like to work on any/all of this?

While I'm generally happy to help out, I'm not eager to do changes to the format implementations. I assume it is much less work for those of you who are already familiar with the C code base.

[solardiz]

@exploide This is your opportunity to become familiar with the C and maybe the OpenCL code base, so that you'd be able to contribute more in the future. ;-)

[reinistihovs]

Hi! Any news on this? Any way to workaround this issue manually?

[solardiz]

@reinistihovs No, and no. You want to contribute a fix? Initially, and that's pretty easy, we need JtR to correctly refuse to process the unsupported inputs. Adding the support would then be the next task.

[reinistihovs]

@solardiz I got it to work, heres my very very dirty fix: change all sha1 occurances to sha256 in: pem_fmt_plug.c and opencl_pem_fmt_plug.c

then Build, pem self test fails.

then used unchanged pem2john.py to create hash.

then used ./john --skip-self-tests test.hash

success:

Using default input encoding: UTF-8
Loaded 1 password hash (PEM, PKCS#8 private key (RSA/DSA/ECDSA) [PBKDF2-SHA256 256/256 AVX2 8x 3DES/AES])
Cost 1 (iteration count) is 4096 for all loaded hashes
Cost 2 (cipher [1=3DES 2/3/4=AES-128/192/256]) is 1 for all loaded hashes
Will run 4 OpenMP threads
Proceeding with single, rules:Single
Press 'q' or Ctrl-C to abort, almost any other key for status
Almost done: Processing the remaining buffered candidate passwords, if any.
Proceeding with wordlist:./password.lst
123456           (?)
1g 0:00:00:00 DONE 2/3 (2021-12-21 13:16) 11.11g/s 5688p/s 5688c/s 5688C/s 123456..crawford
Use the "--show" option to display all of the cracked passwords reliably
Session completed.

[solardiz]

@reinistihovs Hey, that's interesting. I thought it'd require a bit more than that - e.g., also updating the hash size somewhere.

Have you got the OpenCL format to work as well (yes, you updated its source file, but the above doesn't test it)?

Would you submit a proper fix (that would keep things working for SHA-1 as well)?

[reinistihovs]

I would be happy to help, but I am not a experienced C developer, it would take me days to solve this :) I didnt touch any other part of the code, yes the tests FAIL, thats why I used: --skip-self-tests

[solardiz]

@reinistihovs OK, we'll take this from here. Your success so far is encouraging.

[pradkrish]

Hello, I would like to give this a go, if no one else is working on it. It's a lot of new concepts for me, I will be happy to get a break down, so I can solve some simpler problems first before moving on to more complicated ones. This can help me get familiar with the code base.

Notice the sha256 there. For a key generated with older OpenSSL and crackable by current JtR, there's sha1 in that place.

is the first step implementing a check in JtR and refuse to process if it's sha256?

[solardiz]

is the first step implementing a check in JtR and refuse to process if it's sha256?

Yes. You can do it in two ways/steps:

1. Minimal initial: Have pem2john.py report an error (and not produce a "hash") for anything other than sha1.

2. Better suited for further steps: When encountering anything other than sha1, have pem2john.py switch to a new output encoding with this extra field, and have anything other than sha1 initially rejected by the "format" in john proper (perhaps with a warning printed out of its valid function once - search the tree for warned to see some examples).

I think it makes sense to implement the minimal initial change first, and only implement the second when we're about to add the support to john proper as well.

[pradkrish]

Now that the minimal initial is done, I would like to continue working on the next step. As I said before, a lot of new concepts for me here, apologies if it sounds too basic :)

When encountering anything other than sha1, have pem2john.py switch to a new output encoding with this extra field,

Is the encoding for output that is not sha1 already implemented? Or is implementing other encodings part of this task?

and have anything other than sha1 initially rejected by the "format" in john proper (perhaps with a warning printed out of its valid function once - search the tree for warned to see some examples)

I see a bunch of methods in pem_fmt_plug.c and pem_common_plug.c. Specifically, in pem_valid, we seem to check for ciphertext length amongst other things. Is this where the warning should be printed?

[solardiz]

Is the encoding for output that is not sha1 already implemented?

No, but it should be the same as for sha1 except that it'd need to indicate that it's a new one and it should have a field for prf algorithm.

Or is implementing other encodings part of this task?

Yes.

Right now, we output strings that start with $PEM$1. In the non-sha1 case, let's output $PEM$2 instead and follow that by the extra field, so e.g. $PEM$2$sha256, and keep the rest of the string formatting the same.

in pem_valid, we seem to check for ciphertext length amongst other things. Is this where the warning should be printed?

Yes.

[pradkrish]

Is 640 the expected ciphertext length for sha1? If yes, I would love to know why.

[solardiz]

Is 640 the expected ciphertext length for sha1? If yes, I would love to know why.

No, there isn't an "expected ciphertext length for sha1". 640 is a possible one, some others are also possible (see the test vectors we have in pem_common_plug.c), and this isn't directly related to the prf algorithm in use, although there might happen to be a correlation.

[solardiz]

The recent PR said it "partially fixes" this issue, but GitHub doesn't parse the word "partially", so it closed the issue. I'll reopen for us to remember to fix the issue fully.

[pradkrish]

Now I am ready for hopefully the last part of this issue. This might be both interesting and challenging. This is what I gather after snooping around in the code a little. pem_fmt_plug.c calls pbkdf2_sha1(...) and there is an equivalent function pbkdf2_sha256(...). Maybe after parsing what prf encoding has been used, we store it in self.params and call one of the above functions...? I am happy to get a break down.

[solardiz]

@pradkrish Can you just try to research this on your own, without me (or someone) doing it for you? The way the two of us worked on the previous PR isn't practical long-term - it's 90% overhead and 10% progress - so can we already start to improve the work efficiency now? I understand you might still need someone else to take a look and provide guidance on specific sub-issues, but please at least try on your own first? Thank you!

[solardiz]

As we've approached the more serious part of "the project" here, to confidently provide guidance to you I would have needed essentially to do the work, then somehow instead of simply creating a PR myself guide you through arriving at the same. That feels inefficient. And if I start giving advice without fully thinking "the project" through, we'd be going through lots of re-works like we did on the previous PR, which is also inefficient.

[pradkrish]

I agree, the previous PR got overly lengthy, not proud of it. This is a completely new and unrelated field for me, which could be one reason. I will dig around more on my own or pick some other low hanging fruits to learn more about this code base.