search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.27k stars 2.1k forks source link

Make a new release soon as the previous one has Python 3 compatibility issues and known-fixed bugs #4564

Open noraj opened 3 years ago

noraj commented 3 years ago 
$ /usr/bin/env python --version
Python 3.9.1

$ ssh2john id_rsa
Traceback (most recent call last):
  File "/usr/bin/ssh2john", line 193, in <module>
    read_private_key(filename)
  File "/usr/bin/ssh2john", line 103, in read_private_key
    data = base64.decodestring(data)
AttributeError: module 'base64' has no attribute 'decodestring'

Same kind of error as for #3345.

$ grep -n decodestring /usr/bin/ssh2john
103:        data = base64.decodestring(data)

It is preset in 1.9.0-jumbo-1 git tag:

https://github.com/openwall/john/blob/a16c8a76259ab870c07e5123c237b1900402d9a6/run/ssh2john.py#L103

But it's fixed in bleeding-jumbo git tag:

https://github.com/openwall/john/blob/07c0f3c6b00e29bedcfd2ef23cd0c8b39888c7be/run/ssh2john.py#L107-L114

tag 1.9.0-jumbo is from May 14, 2019.

So releasing a new version will help fix that bug in distro packages that are mostly using python 3.9 nowadays.

System configuration

$ john --list=build-info                                                                         
Version: 1.9.0-jumbo-1
Build: linux-gnu 64-bit x86_64 AVX AC MPI + OMP
SIMD: AVX, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
System-wide exec: /usr/lib/john
System-wide home: /usr/share/john
Private home: ~/.john
CPU tests: AVX
CPU fallback binary: john-non-avx
$JOHN is /usr/share/john/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 10.2.0
GNU libc version: 2.32 (loaded: 2.32)
OpenCL headers version: 2.2
Crypto library: OpenSSL
OpenSSL library version: 01010108f      (loaded: 01010109f)
OpenSSL 1.1.1h  22 Sep 2020     (loaded: OpenSSL 1.1.1i  8 Dec 2020)
GMP library version: 6.2.0      (loaded: 6.2.1)
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's

Workaround for distro packager waiting for the new release

$ sed 's/decodestring/decodebytes/' /usr/bin/ssh2john | python3.9 - id_rsa

ArchLinux FS#69545

anthraxx commented 3 years ago

I would highly appreciate to get a guicy release :P

solardiz commented 3 years ago

@noraj Do you want to help us make that release sooner rather than later? We need to at least bring NEWS up to date (many important changes were unfortunately not reflected in there yet) and try and fix all of the issues under the "Definitely 1.9.0-jumbo-2" milestone. Want to help with these?

noraj commented 3 years ago

@solardiz You mean that some major changes where not documented into NEWS? How could I help?

solardiz commented 3 years ago

@noraj Right. The task is to go over the commits made since 1.9.0-jumbo-1 and add to NEWS non-trivial higher-level changes describing them in a suitable form (see existing entries), and send a pull request updating NEWS. Thank you!

solardiz commented 3 years ago

I've just removed most issues from the "Definitely 1.9.0-jumbo-2" milestone. Most of those went to the "Potentially 1.9.0-jumbo-2" milestone, some were removed from milestones altogether. Some of the issues only need more review to determine that we can close them or don't have to work on them before the release. Some need actual work.

solardiz commented 3 years ago

go over the commits made since 1.9.0-jumbo-1 and add to NEWS non-trivial higher-level changes

This task is now #4570.

We also had #3979 titled after for the upcoming release, but it has some "irrelevant" stuff said on it (on general and longer-term planning rather than on what really needs to be done for the release).

So I suggest we use this new issue for planning the release from now on.

claudioandre-br commented 3 years ago

Another example of a "bug" fixed in bleeding-jumbo:

$ john teste.hash 
Using default input encoding: UTF-8
Loaded 1 password hash (HMAC-SHA256 [password is key, SHA256 256/256 AVX2 8x])
Will run 8 OpenMP threads
Proceeding with single, rules:Single
Press 'q' or Ctrl-C to abort, almost any other key for status
Almost done: Processing the remaining buffered candidate passwords, if any.
Proceeding with wordlist:/snap/john-the-ripper/current/run/password.lst
Proceeding with incremental:ASCII
0g 0:00:00:12  3/3 0g/s 2068Kp/s 2068Kc/s 2068KC/s licuato..lynns17
Session aborted

$ cat teste.hash 
john.txt
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa#26a8d8c86b34a358817ced22bfde859642ba4d13a30a0b814345cb3e878b8618
solardiz commented 3 years ago

@claudioandre-br No, this wasn't anything we fixed - rather, the line wrapping prevented you from testing the same thing that the question on john-users was about. Anyway, let's not collect supposedly fixed issues in here. (If you feel anything like this should be added to NEWS, we have a separate issue for updating that file.) I'd rather see us make progress towards making a new release. Thanks!

D3vil0p3r commented 1 year ago

Since on OpenWall website the latest john package is still the one with this issue, a new working official release with these fixes will be released or not? Or we should use the bleeding version forever? I see that the latest version on the website is on 2019...

solardiz commented 1 year ago

@D3vil0p3r A new release will be made. Hopefully soon, but no specific date has been set. We should be making releases more often. Unfortunately, releases were also infrequent before - e.g., 1.8.0-jumbo-1 in December 2014 followed by 1.9.0-jumbo-1 only in May 2019.

D3vil0p3r commented 1 year ago

Thank you for the answer @solardiz ! The important aspect for me, even if there is no a date, in the future could be a new release in order to fix the described issue with the latest "stable" official release. Thank you again ^^

ryandesign commented 1 year ago

Please make a new release. Your latest release does not compile on Apple Silicon Macs. You have already fixed this problem in the code years ago (#4585) but users continue to encounter it as they attempt to use your latest release.

ryandesign commented 1 year ago

Your latest release does not compile on Intel Macs either due to multiple definition of symbols, another problem you fixed in the repository years ago (#4258).

noraj commented 9 months ago

Another bug already been fixed on bleeding jumbo

➜ ssh2john ~/.ssh/id_ed25519_crack
Traceback (most recent call last):
  File "/usr/bin/ssh2john", line 194, in <module>
    read_private_key(filename)
  File "/usr/bin/ssh2john", line 154, in read_private_key
    saltstr = data[salt_offset:salt_offset+salt_length].encode("hex")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'bytes' object has no attribute 'encode'. Did you mean: 'decode'?
➜ python --version
Python 3.11.6
prbhtkumr commented 6 months ago

Another bug already been fixed on bleeding jumbo

➜ ssh2john ~/.ssh/id_ed25519_crack
Traceback (most recent call last):
  File "/usr/bin/ssh2john", line 194, in <module>
    read_private_key(filename)
  File "/usr/bin/ssh2john", line 154, in read_private_key
    saltstr = data[salt_offset:salt_offset+salt_length].encode("hex")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'bytes' object has no attribute 'encode'. Did you mean: 'decode'?
➜ python --version
Python 3.11.6

I've made a fix locally by using the hex() method instead of encode("hex"):

saltstr = data[salt_offset:salt_offset+salt_length].hex()

I believe this change should resolve the issue. However, I'm not sure where in the codebase this change should be made to reflect it correctly. Any guidance on this would be appreciated.

solardiz commented 6 months ago

@prbhtkumr My understanding is that all bugs discussed on this issue are already fixed in this repo. The only reason you ran into those bugs is you're using an older version. We appreciate your willingness to contribute fixes, but for that please start by using our latest code from here to find/fix bugs that still exist. Thank you!

claudioandre-br commented 4 months ago

Early and Often

Thinking out loud:

Therefore (or because of this):

[1] rc or ce or ... Any "flag" indicating the final version is not ready yet.

solardiz commented 4 months ago

I don't mind us starting to create 1.9.1, etc. releases on GitHub while preparing for a 2.0.0 release from Openwall, but I also don't intend to spend my time on it currently. Claudio, is this something you'd like to start doing?

claudioandre-br commented 4 months ago

I don't mind doing that. But I simply don't have the power to do it. Using a GUI or CLI takes more than I can do.

The release commit itself is very simple, but I can create a PR. I won't dwell too much on release notes and things like that.

"This is a bug fix, renovate, and update release intended to provide a modernized version to end users and packagers. "Early and Often".

BTW: I'll pick a commit, change the release ID and such. And that's it. After that, I will go back to using the bleeding name. GitHub itself will be the advertising platform.

Just to be clear, only one thing bothers me: it is #3517. It seems like a change in behavior that is not clear whether it is right or wrong.