search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.27k stars 2.1k forks source link

1.9.0-jumbo-1 #3513

Closed magnumripper closed 5 years ago

magnumripper commented 5 years ago

Lets's list / discuss here what we need to do before a release. Or rather, a list of NEED, one of NICE and one of DON'Ts.

See also #1879

magnumripper commented 5 years ago

I'm sorry I can't help with more testing or relbench checks. It's sad we're so short of volunteers (and strange! Sometimes I wonder if it's my fault?)

magnumripper commented 5 years ago

Hilarious. I got a mail from CircleCI where they apologised for having been slow lately - and I thought they were the quick ones! 🙃

claudioandre-br commented 5 years ago

CircleCI had problems (and I think is nice they acknowledged that).

On the other hand, Travis is a victim of its own success. (Comparing with 2 years ago) we are running on worse hardware.

solardiz commented 5 years ago

I'm testing on various archs now. Got a failure and a pending fix for "ansible" on archs that require alignment. Got a not-yet-investigated failure for "monero" on big-endian.

solardiz commented 5 years ago

Also got a failure and a pending fix for "phpass" on 64-bit archs that require alignment. I wonder how that one bug could have persisted that long.

magnumripper commented 5 years ago

Had a quick look at monero. Maybe I'm too tired but I see absolutely nothing that could be a problem on BE.

magnumripper commented 5 years ago

...unless the chacha code has problems of course. But the keepass format worked fine? It's just those two using chacha.

solardiz commented 5 years ago

I also have no luck with "monero" so far. But I've got a fail+fix for "racf-kdfaes".

magnumripper commented 5 years ago

What was the fail in monero? cmp_all?

solardiz commented 5 years ago 
$ OMP_NUM_THREADS=1 ../run/john -test -form=monero
Warning: OpenMP is disabled; a non-OpenMP build may be faster
Benchmarking: monero, monero Wallet [Pseudo-AES / ChaCha / Various 64/64]... FAILED (cmp_all(1))
magnumripper commented 5 years ago

Problem should be in slow_hash_plug.c

solardiz commented 5 years ago

Maybe. Meanwhile, I'm working on some documentation updates for core.

magnumripper commented 5 years ago

Problem should be in slow_hash_plug.c

Hmmm or any of the gazillion hashes it uses. Perhaps we should just disable monero for BE.

solardiz commented 5 years ago

Perhaps we should just disable monero for BE.

I'm OK with that.

magnumripper commented 5 years ago

IIRC I have a really crappy sparc somewhere, I'll see if I can even find it in time (let alone boot it).

magnumripper commented 5 years ago

@claudioandre-br don't you run lots of BE tests regularly?

solardiz commented 5 years ago

Also found/fixed two kinds of alignment issues in "wpapsk".

claudioandre-br commented 5 years ago

3438 (as an example)

magnumripper commented 5 years ago

Ah, yes. We just ignored it 😢

solardiz commented 5 years ago

@magnumripper 1.9.0 core is ready, please merge. Thanks!

magnumripper commented 5 years ago

That was an easy one

magnumripper commented 5 years ago

When you say "ready" I was expecting params.h to say "1.9.0"

solardiz commented 5 years ago

Core's params.h does say 1.9.0 for me.

magnumripper commented 5 years ago

Now I see it!

solardiz commented 5 years ago

@jfoug formspring_fmt_plug.c has:

#define ALGORITHM_NAME          "?" /* filled in by dynamic */

and I'm seeing this moderate weirdness:

Testing: FormSpring [sha256($s.$p) 64/64sha2-OpenSSL]... PASS

I mean the "sha2-OpenSSL" string, which appears right after "64/64" without a space. Perhaps fix this? I think we need a space instead of the "sha2-" string.

magnumripper commented 5 years ago

I'll have a look at that.

Please do not forget this when eventually baking a jumbo release tarball

diff --git a/src/params.h b/src/params.h
index 290aef16d..0c38ebbd0 100644
--- a/src/params.h
+++ b/src/params.h
@@ -31,13 +31,13 @@
 /*
  * Define this for release tarballs. It affects the version reporting (will
  * be the string above and below and never a Git hash) as well as some other
  * details. Eg. it mutes output of OpenCL run-time build log unless the build
  * failed.
  */
-//#define JTR_RELEASE_BUILD 1
+#define JTR_RELEASE_BUILD 1

 /*
  * Jumbo's version number. Note that we must uncomment JTR_RELEASE_BUILD
  * above, in any release tar-balls (and only then, never ever in Git).
  */
 #define JUMBO_POSTFIX                  "-jumbo-1"
solardiz commented 5 years ago

More of the same weirdness:

Testing: FormSpring [sha256($s.$p) 64/64sha2-OpenSSL]... PASS
Testing: hMailServer [sha256($s.$p) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_50 [sha224($p) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_60 [sha256($p) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_61 [sha256($s.$p) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_70 [sha384($p) 64/64 sha2-OpenSSL]... PASS
Testing: dynamic_80 [sha512($p) 64/64 sha2-OpenSSL]... PASS
Testing: dynamic_1029 [sha256($p) (hash truncated to length 32) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_1503 [sha256(sha256($p).$s) (XenForo SHA-256) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_1528 [sha256($s.$p.$s) (Telegram for Android) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_1588 [sha256($s.sha1($p)) (ColdFusion 11) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_1602 [sha256(#.$salt.-.$pass) (QAS vas_auth) 64/64sha2-OpenSSL]... PASS
Testing: dynamic_1608 [sha256(sha256_raw(sha256_raw($p))) (Neo Wallet) 64/64sha2-OpenSSL]... PASS

Algorithm names intermixed with implementation bitness:

Testing: tc_aes_xts, TrueCrypt AES256_XTS [SHA512 64/64 /RIPEMD160/WHIRLPOOL]... (33xOMP) PASS
solardiz commented 5 years ago

BTW, "64/64" is also wrong for scalar SHA-256 and SHA-224. It's 32-bit only, so should be "32/32" or "32/64".

magnumripper commented 5 years ago

Doh. I'll find it.

solardiz commented 5 years ago

More SHA-256 misreported as 64-bit:

Testing: ansible, Ansible Vault [PBKDF2-SHA256 HMAC-256 64/64 OpenSSL]... (33xOMP) PASS
Testing: AzureAD [PBKDF2-SHA256 64/64 OpenSSL]... (33xOMP) PASS
Testing: BitLocker, BitLocker [SHA-256 AES 64/64]... (33xOMP) PASS
Testing: Bitwarden, Bitwarden Password Manager [PBKDF2-SHA256 AES 64/64 OpenSSL]... (33xOMP) PASS
Testing: FVDE, FileVault 2 [PBKDF2-SHA256 AES 64/64 OpenSSL]... (33xOMP) PASS
Testing: keyring, GNOME Keyring [SHA256 AES 64/64 OpenSSL]... (33xOMP) PASS
Testing: notes, Apple Notes [PBKDF2-SHA256 AES 64/64 OpenSSL]... (33xOMP) PASS
Testing: Padlock [PBKDF2-SHA256 AES 64/64 OpenSSL]... (33xOMP) PASS
Testing: PBKDF2-HMAC-SHA256 [PBKDF2-SHA256 64/64 OpenSSL]... (33xOMP) PASS
Testing: pwsafe, Password Safe [SHA256 64/64 OpenSSL]... (33xOMP) PASS
Testing: RAR5 [PBKDF2-SHA256 64/64 OpenSSL]... (33xOMP) PASS
Testing: SybaseASE, Sybase ASE [SHA256 64/64 OpenSSL]... (33xOMP) PASS
Testing: vdi, VirtualBox-VDI AES_XTS [PBKDF2-SHA256 64/64 OpenSSL + AES_XTS]... (33xOMP) PASS
Testing: HMAC-SHA256 [password is key, SHA256 64/64 OpenSSL]... (33xOMP) PASS

Some of these might refer to AES (does it use 64-bit?), but when there's PBKDF2-SHA256 we should only report on properties of that slow step, not on whatever fast post-processing there might be.

IIRC, Whirlpool is also 32-bit only:

Testing: tc_whirlpool, TrueCrypt AES256_XTS [WHIRLPOOL 64/64]... (33xOMP) PASS

Scalar SHA-1 misreported as 32-bit:

Testing: as400-ssha1, AS400-SaltedSHA1 [sha1(utf16be(space_pad_10(uc($s)).$p)) (IBM AS/400 SHA1) 64/64]... PASS
Testing: sha1crypt, NetBSD's sha1crypt [PBKDF1-SHA1 64/64]... (33xOMP) PASS
Testing: dynamic_1590 [sha1(utf16be(space_pad_10(uc($s)).$p)) (IBM AS/400 SHA1) 64/64]... PASS

Duplicate "DES DES" (but 64/64 is possibly OK, if bitslice):

Testing: mschapv2-naive, MSCHAPv2 C/R [MD4 DES DES 64/64 naive]... (33xOMP) PASS
magnumripper commented 5 years ago

What about "gost, GOST R 34.11-94 [64/64]" is that correct?

solardiz commented 5 years ago

I don't know. It might be.

magnumripper commented 5 years ago

I'm done now, need sleep. Good luck, my friend.

magnumripper commented 5 years ago

What about "gost, GOST R 34.11-94 [64/64]" is that correct?

It is.

BTW cba22ea9 fixed lots of other algo names but commit message accidentally referred to €3513 instead of #3513

magnumripper commented 5 years ago

I really don't have more time now, for anything. Will commit a few minor things I found, then I'm gone. TTFN

magnumripper commented 5 years ago

I will check later tonight for any core patches though

magnumripper commented 5 years ago

I temporarily dropped the requirement for PR reviews. Use wisely, eg. if you merely fixed a typo then don't bother waiting for bots or reviews, just merge

solardiz commented 5 years ago

I'm in the process of releasing 1.9.0 core. It's already on the website, and I'm about to write its announcement. I'm going to say that these days it's primarily just core for jumbo and that a corresponding jumbo release is coming shortly.

solardiz commented 5 years ago

1.9.0 core is fully out and announced. I'm likely to proceed with 1.9.0-jumbo-1 tomorrow or later - am tired now, and there are still minor changes to make and non-trivial announcement detail to write.

magnumripper commented 5 years ago

The following shows the number of Jumbo commits since 1.8.0-jumbo-1, not counting the ones merged from core and not counting other merge commits (i.e. we do count non-core commits that were merged, just not the merge "glue" commit itself).

$ git log --no-merges 1.8.0-jumbo-1..HEAD ^origin/master --pretty=oneline | wc -l
    5904

A diffstat is too long to post here...

$ git diff --stat 1.8.0-jumbo-1..HEAD | wc -l
    1999

But it ends like this:

 1998 files changed, 823800 insertions(+), 192254 deletions(-)

I'm not quite sure, but doesn't this mean we could say "over a million changes"? 👀 😃

magnumripper commented 5 years ago

Here's a full changelog (it's also in the tree as doc/CHANGES-jumbo.git)

magnumripper commented 5 years ago

@solardiz feel free to re-word anything here. I'm just throwing out things that pop up.

Just listing some highlights. This is random things that pop up in my head and that I see when surfing randomly in the 6,000 commit descriptions:

(moved to wiki: https://github.com/magnumripper/JohnTheRipper/wiki/1.9.0-jumbo-1-NEWS)

I have to quit and get some sleep. I was using git log --color=always --no-merges 1.8.0-jumbo-1..HEAD ^origin/master --oneline | tac | less -r and stopped at 55fde2fcc which apparently is beginning of 2016. More than three years more to browse.

solardiz commented 5 years ago

On a FreeBSD system, I have to explicitly use gmake instead of make after running our ./configure. Do we have this requirement of GNU make documented anywhere?

solardiz commented 5 years ago

I've proceeded to post my relbench results to #2914.

solardiz commented 5 years ago

I've just created issues #3811 #3812 #3813 #3814 #3815 #3816 #3817 #3818 for the 10%+ performance regressions.

solardiz commented 5 years ago

I have to quit and get some sleep. I [...] stopped at 55fde2f which apparently is beginning of 2016. More than three years more to browse.

1.8.0-jumbo-1 is from December 2014 (we released it much later than 1.8.0 core), so it's just 1+ year more to browse. Anyone else willing to help, now that magnum is away?

claudioandre-br commented 5 years ago

Another highlight.

magnumripper commented 5 years ago

I’ll be back in business here on Sunday April 28th, evening CET. If Jumbo isn’t yet released by then, I’ll spend all available time on it until it is.

solardiz commented 5 years ago

Our jumbo-specific README.md still has the partial and non-updated yet lengthy list of jumbo-specific documentation files, which magnum agreed we should simply drop from there. I'd like to see that happen, maybe along with other pre-release edits/cleanups of that file. Claudio?

solardiz commented 5 years ago

Claudio took care of some cleanup of README.md. Thanks!

magnum, now that you're back perhaps you can continue preparing the release highlights list? Maybe put it in a NEWS file or such in the repo, so that we'll hopefully maintain it going forward. Also, since we release so infrequently maybe group those changes not only by release (for now that would be just one list of all changes since 1.8.0-jumbo-1), but also by year. Thanks!