-internal Current failures

[jfoug]

Start working on these. Some we will fix jtrts.pl (list when we do), and some we will fix the format. Some we may just accept the problem,, but at least list why.

NOTE, this list updated with cpu AFTER 9a46ca2 was applied.

$ ./jtrts.pl -internal -q -case_mangle cpu  ***** ALL FIXED *****

here is the GPU failures from my laptop (NOTE some are driver fails)

$ ./jtrts.pl -internal -case_mangle gpu
-------------------------------------------------------------------------------
- JtR-TestSuite (jtrts). Version 1.13, Dec 21, 2014.  By, Jim Fougeron & others
- Testing:  John the Ripper password cracker, version 1.8.0.2-jumbo-1-bleeding_omp [cygwin 64-bit XOP-autoconf]
--------------------------------------------------------------------------------

ALL of these fail ST on my laptop card.
descrypt-opencl           guesses:    0 -show=   0 0:00:00:00 DONE : Expected (6)(-show6)
pbkdf2-hmac-sha512-opencl guesses:    0 -show=   0 0:00:00:38 DONE : Expected (9)(-show27)
sha256crypt-opencl        guesses:    0 -show=   0 0:00:00:01 DONE : Expected (4)(-show4)
sha512crypt-opencl        guesses:    0 -show=   0 0:00:00:01 DONE : Expected (5)(-show5)

krb5pg-md5 fails ST on get_hash[0](3)  The only reason it finds 6 (vs 9), is because we are using -skip
krb5pa-md5-opencl         guesses:    6 -show=  21 0:00:00:00 DONE : Expected (9)(-show27)

oldoffice-opencl  fails ST on cmp_all(1)  The only reason it finds 6 (vs 9), is because we are using -skip
oldoffice-opencl          guesses:    1 -show=   3 0:00:00:00 DONE : Expected (5)(-show15)

gpg crashed my driver
gpg-opencl                guesses:    0 -show=   0 unk unk : Expected (6)(-show6)

Fixed split() function (for tagged hashes)
raw-sha512-opencl         guesses:   13 -show=  33 0:00:00:00 DONE : Expected (12)(-show36)

These were fixed by making them FMT_SPLIT_UNI formats

form=skein-256                    guesses:    1 0:00:00:00 DONE  [PASSED]
form=skein-512                    guesses:    2 0:00:00:00 DONE  [PASSED]
form=tiger                        guesses:    2 0:00:00:00 DONE  [PASSED]
form=whirlpool                    guesses:    1 0:00:00:00 DONE  [PASSED]
form=whirlpool0                   guesses:    1 0:00:00:00 DONE  [PASSED]
form=whirlpool1                   guesses:    1 0:00:00:00 DONE  [PASSED]

These were fixed by making non proper cased hashes fail valid()

[jfoug]

skien, tiger and whirlpool are all raw formats, so I will simply add a split to them and be done with it.

[magnumripper]

What version does --list=opencl-devices report? And does it work better with --dev=cpu?

[jfoug]

$ ../run/john -list=opencl-devices
Platform #0 name: AMD Accelerated Parallel Processing
Platform version: OpenCL 1.2 AMD-APP (1348.5)
        Device #0 (0) name:     Devastator
        Board name:             AMD Radeon HD 7640G
        Device vendor:          Advanced Micro Devices, Inc.
        Device type:            GPU (LE)
        Device version:         OpenCL 1.2 AMD-APP (1348.5)
        Driver version:         1348.5 (VM)
        Native vector widths:   char 16, short 8, int 4, long 2
        Preferred vector width: char 16, short 8, int 4, long 2
        Global Memory:          765.7 MB
        Local Memory:           32.0 KB (Local)
        Max memory alloc. size: 191.1 MB
        Max clock (MHz):        654
        Profiling timer res.:   1 ns
        Max Work Group Size:    256
        Parallel compute cores: 4
        Stream processors:      256  (4 x 64)
        PCI device topology:    00:01.0

        Device #1 (1) name:     AMD A8-4500M APU with Radeon(tm) HD Graphics
        Device vendor:          AuthenticAMD
        Device type:            CPU (LE)
        Device version:         OpenCL 1.2 AMD-APP (1348.5)
        Driver version:         1348.5 (sse2,avx,fma4)
        Native vector widths:   char 16, short 8, int 4, long 2
        Preferred vector width: char 16, short 8, int 4, long 2
        Global Memory:          5.0 GB
        Global Memory Cache:    16.0 KB
        Local Memory:           32.0 KB (Global)
        Max memory alloc. size: 2.0 GB
        Max clock (MHz):        1896
        Profiling timer res.:   539 ns
        Max Work Group Size:    1024
        Parallel compute cores: 4
        PCI device topology:    00:00.0

[magnumripper]

That is a very old driver, you might want to update it. Try adding --pass=--dev=cpu and see what happens.

[magnumripper]

I get this now, without case mangle.

form=pbkdf2-hmac-sha256-opencl    guesses:   11 -show=  11 0:00:00:05 DONE : Expected count(s) (23)(-show23)  [!!!FAILED!!!]
.pot CHK:pbkdf2-hmac-sha256-openc guesses:   11 0:00:00:06 DONE  [PASSED]

That's the only problem without case mangling. It looks like an actual bug to me so far. For some reason I did not get this some day ago. I wonder why.

[jfoug]

Running now.

Btw, I now have a version of jtrts.pl where i believe I have -case_mangle running 100% correct, with all formats handled.

Note, if I do not FIND a hex field, that is of length:

if ($len == 16 || $len == 32 || $len == 40 || $len == 48 || $len == 56 || $len == 64 ||
    $len == 80 || $len == 96 || $len == 104 || $len == 112 || $len == 128) {

then I do not output ANY string. There are 3 formats where I break that rule. Those formats are FMT_SPLIT_UNI formats, BUT contain some base-64 stuff. Those are pbkdf2-hmac-sha1 (contains pkcs5 stuff), and raw-sha256[-ng] which both have cisco4 hashes. So I simply put some 'magic' in there right before I return an empty string, and if I detect one of those signatures, I then return the hash and a \n so that the -salt=count*3 logic will work.

Now, if it is a fmt_split_uni, I check for $cnt and show=$cnt*3 only. For non-splits, I check for $cnt -show=$cnt ONLY. This caught a few more issues (1 in a format, several in jtrts.pl). I do think the logic for -case_mangle is now solid, with only those 2(3) exceptional formats being worked around.

I will get jtrts.pl uploaded when I get my test run fully completed, and am assured that ALL issues have been found.

GOOD stuff. We now have much better cleaned up of split/valid, and we found about 20-30 formats that needed some work, and even found some that were totally busted.

[magnumripper]

AMD's CPU driver is almost always free of bugs so it's good for verifying things.

Note, I committed some minor fixes ten minutes ago.

[magnumripper]

I will get jtrts.pl uploaded when I get my test run fully completed, and am assured that ALL issues have been found.

Am I now wasting time on a buggy version?

[jfoug]

no, it is still fine. If there are issues, then only the ones that fail will need a re-test.

[jfoug]

Let me check in what i have. I have 1 off tested all CPU, and the GPU that I can test without the format having errors also all passed.

[magnumripper]

With -case_mangle I still only see that one problem with pbkdf2-hmac-sha256-opencl (only tested OpenCL formats).

[magnumripper]

BTW it's Base64 so not affected by case mangling.

[magnumripper]

Oh, and exact same problem seen with the CPU format.

[magnumripper]

LOL, it even fails self-test. I think I need to "make clean" after some of your recent commits in bleeding.

[jfoug]

The new fix will fix that. the base-64 stuff. In those cases, the script MUST triple print those base-64 lines.

[magnumripper]

I think I need to "make clean" after some of your recent commits in bleeding.

That was it. Our Makefile is sloppy so it doesn't ensure a rebuild of a format when a shared header was changed. That should actually be a new issue (in bleeding, not here)

[jfoug]

$ ../run/john -ses=./tst  --dev=cpu -pot=./tst.pot selftest.in --wordlist=selftest.dic -form=descrypt-opencl
Device 1: AMD A8-4500M APU with Radeon(tm) HD Graphics
Local worksize (LWS) 64, Global worksize (GWS) 16384
Loaded 6 password hashes with 4 different salts (descrypt-opencl, traditional crypt(3) [DES OpenCL])
Self test failed (get_hash[0](0))

$ ../run/john -ses=./tst -pot=./tst.pot selftest.in --wordlist=selftest.dic -form=descrypt-opencl
Device 0: Devastator (AMD Radeon HD 7640G)
Local worksize (LWS) 64, Global worksize (GWS) 16384
Loaded 6 password hashes with 4 different salts (descrypt-opencl, traditional crypt(3) [DES OpenCL])
Self test failed (get_hash[0](0))

I am seeing same failures for both the GPU and CPU code.

[magnumripper]

That's odd, their CPU devices usually are pretty much free from bugs. Try upgrading.

[jfoug]

That was it. Our Makefile is sloppy so it doesn't ensure a rebuild of a format when a shared header was changed. That should actually be a new issue (in bleeding, not here)

That was what I was mentioning the other day, about another BAD side effect of putting code into headers. This change will be HARD to do, based upon a lot of our makefile is generated by *_plug.c Having additional dependencies will require some other magic. If all that was properly in source files, we would not have this issue. There are very few places (but I give you there are some), where the inlining means shit for most of that .h code crap. Most of it is just bad practices, vs having proper external functions.

[jfoug]

That should actually be a new issue (in bleeding, not here)

https://github.com/magnumripper/JohnTheRipper/issues/955

[magnumripper]

All CUDA formats pass, even with case mangling.

[magnumripper]

For good measure I tested CPU formats too and all pass. So we have zero problems.

[jfoug]

Good. Closing this issue, AND the Alternate test. This was a very good addition. It took a while to get both the TS and JtR to the same goal, but now that we are there, I am very happy you suggested this new mode.

[magnumripper]

Yeah I think we found quite a few more or less bad bugs! Good stuff.