search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.29k stars 2.1k forks source link

SIMD/NonSIMD formats porting to BE systems #2888

Closed jfoug closed 6 years ago

jfoug commented 6 years ago

Now that we have availability to BE SIMD systems (power64 AltiVec), all of the SIMD enhanced formats will require porting to properly work for the BE system. The SIMD code up to this point, was specially crafted to place input into proper format, and compare binary vs output, BUT it was only correct for LE systems.

Non-SIMD Noted in #2861

There will be a list (below) of formats needing updated. Check them off one by one when completed. To be complete, make sure:

  1. continues to work with no speed reduction in LE SIMD (both OMP and non-OMP)
  2. continues to work with no speed reduction in LE non-SIMD (both OMP and non-OMP)
  3. continues to work with no speed reduction in BE non-SIMD (both OMP and non-OMP)
  4. Works in BE SIMD builds (both OMP and non-OMP)
  5. NOTE, be SURE to check -enc=utf8 and -enc=cp737 to flush out encode using different key loaders.
jfoug commented 6 years ago

I now have examples of code for both LE and BE formats (MD4/5 and SHA1). The code in the get_key is about the same. The difference is BE formats do not require swapping in the getkey like the LE versions did, and then the swapping in cmp*() functions only needs done for formats when dealing with algorigthms which are different endianity (i.e. MDx needs byte swapped on BE computers, SHA needs byte swapped on LE computers)

jfoug commented 6 years ago

Wonderful finding!!! pbdkf2-hmac-md5/md5/sh256/sha512 were already properly working in BE. these are 'include' files, so them working means many formats already do work. The change for pbkdf2-sha1 was trivial, seeing the changes in the other pbkdf2-* include files. Getting pbkdf2-hmac-sha1 fixed will get a lot of broken formats working just fine.

kholia commented 6 years ago

I am trying to test these changes on my Debian 8.x PPC64 VM. I ran ./configure which resulted in,

Configured for building John the Ripper jumbo:

Target CPU ................................. powerpc64, 32-bit BE
AES-NI support ............................. no
Target OS .................................. linux-gnu
Cross compiling ............................ no
Legacy arch header ......................... ppc64.h

Running make is failing with the following error,

gcc -DAC_BUILT  -DJOHN_powerpc64 -c -g -O2 -I/usr/local/include -DARCH_LITTLE_ENDIAN=0   -Wall -Wdeclaration-after-statement -fomit-frame-pointer --param allow-store-data-races=0 -Wno-deprecated-declarations  -Wunused-but-set-variable -std=gnu89 -Wdate-time -D_POSIX_SOURCE -D_GNU_SOURCE -D_XOPEN_SOURCE=600 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE   -fopenmp  -pthread   -I/usr/local/include -funroll-loops MD5_fmt.c -o MD5_fmt.o
MD5_fmt.c:389:10: error: expected ‘}’ before ‘BITS’
   "MD5 " MD5_ALGORITHM_NAME,
          ^
MD5_fmt.c:52:25: warning: ‘tests’ defined but not used [-Wunused-variable]
 static struct fmt_tests tests[] = {
                         ^
Makefile:1642: recipe for target 'MD5_fmt.o' failed
make[1]: *** [MD5_fmt.o] Error 1

I wonder how @jfoug went past these compilation error(s).

jfoug commented 6 years ago

I wonder how @jfoug went past these compilation error(s).

by doing this rm arch.h ln -s ppc32.h arch.h make -s clean && make -sj8

It will fix your problem. your build is half pregnant ;) Configure is not right (yet) for these machines (vms)

I also installed the 64 bit tools, build oSSL in 64 bits (I could not find it on apt-get land). And am able to build AltiVec mode. NOTE, there were a LOT more hand editing during configure / build needed to get that AltiVec there. For that env, I am making sure that the 'build' environment I do have is not damaged (build wise), while I am working on getting all the SIMD formats that fail ported over to BE. Some ARE working, and many of the ones using PBKDF2-HMAC-* work!! because the headers were written under hopes of a BE SIMD, and surprisingly that code was right, lol. But I want to make SURE I keep that build env live, and then when done porting, I will get configure to build it properly, and be able to auto-detect better.

jfoug commented 6 years ago

For building 64 bit in that PPC64 VM, I documented (post documented from memory, so the steps are probably not all needed) in post https://github.com/magnumripper/JohnTheRipper/issues/2861#issuecomment-343956298

jfoug commented 6 years ago

Updated dynamic to FORCE it to use non-SIMD on a BE machine. This fixed almost all dynamic and thin dynamic hashes. The only ones left are:

dynamic_9 mediawiki dynamic_compiler (this should have been fixed, but something still seems wrong.

NOTE, I will create a new issue specifically targeted on dynamic format getting it to work with BE SIMD code. The fix here simply gets john to run, without FIXING the very complex dynamic format.

jfoug commented 6 years ago

After getting all of the dynamic thing formats marked as being done, we are down to 48 SIMD formats failing on BE SIMD. Getting there ;)

jfoug commented 6 years ago

Made changest to the grandaddy file (simd-intrinsics.c) lots of changes there. Mostly for flat-in and flat-out, but also for md5crypt (not sure WHY md5crypt code is in this file vs in the fmt file?) In the flat-in and flat-out, we need to do the opposite of what we WERE doing (for all the LE code). So when doing MD4/5 on the BE system, we need to swap (when loading from flat, or saving to flat). Then on sha1/256/512, we need to stop doing the swapping when loading/storing to flat. Those changes were needed to get any format using FLAT simd to work. Once they were done, I got the *crypt formats all working pretty easily.

jfoug commented 6 years ago

Down to 37. One thing of note, when you want to get an 'aligned' uint32_t using the GETPOS, you have to do it a bit differently:

#if ARCH_LITTLE_ENDIAN==1
    uint32_t *kb = (uint32_t*)&saved_key[GETPOS(0, ti)];
#else
    uint32_t *kb = (uint32_t*)&saved_key[GETPOS(3, ti)];
#endif

In both instances above, kb will point to the 'proper' and aligned uint32_t within saved key. Seems strange, until you look at where the actual bytes are in the data structure. Then it makes sense. This is only since we have 2 different variants of the GETPOS macro, which the proper one for the endianity gets compiled in.

Also note, this was on a LE format (where the LE GETPOS was used to get pos 0). If this had been a BE format, I am not sure if the '3' would need to be on a !ARCH_LITTLE_ENDIAN part or in the LITTLE_ENDIAN==1 part. When I get to one, I will know.

jfoug commented 6 years ago

WooHOO! Down to only 20 left. Some (sap*) will also need corrected for alignment problems. Simplest to just do in 1 patch #2868

jfoug commented 6 years ago

I have MSCHAPv2 and netntlm working, but I want to spend more time figuring out if there is a better way. That format SUX!!!!! The binary is in 3 layouts / endianities. There are 2 bytes leading. each are viewed by themselvs, and also viewed as an unsigned short. Then there are the 4 byte blocks (minus the first 2 bytes). So, dealing with endianity flipping on this format was NOT easy. as I had to pull sub parts of 4 byte integers, deal with endianity of where the short lives, and then endianity of the short itself. VERY ugly shit!

It is working now, but there is likely a lot more swapping than needed. I will try to reduce as much as possible. There are other formats i have started on,and hit my head on the wall. Things like wpapsk (SUCK). HDAA (poorly written, so i just have to figure it out). SapB (sucks). SunMD5 (sucks).

I will try to get the mschapv2 done, then start ticking off some others. the ones I am stuck on, I will pick up when I get more motivated ;)

jfoug commented 6 years ago

mschapv2 and netntlm done and checked in. Harder format to port well :(

jfoug commented 6 years ago

@magnumripper some of the ones left are nightmares to BE port. They may simply get SIMD turned off for any BE build (for now), and then leave an open bug to get SIMD working on them at a later date.

This was what was done with Dynamic. That format will be HELLA hard to get ported to BE, and may not be worth it in the end. Currently dyna works just fine, you just do not get AltiVec instructions.

The ones I am having problems with (but think that they 'should' be ported, are: wpapsk, and probably the sapX. SunMD5 is a dead hash (only resurected for CMIYC). HDAA is not really important, and it could live forever as non SIMD for BE. There are other harder formats I have not started on (I picked off the ones I thought easier as I went). I am pretty sure office/NT/phpass/rar all should be done. These may be hard, they may be trivial.

I may need help on some of these.

magnumripper commented 6 years ago

They may simply get SIMD turned off for any BE build (for now)

I'm fine with that, even for a release.

I will build up some steam tomorrow, just catching up now. But I'm not likely to focus on BE/alignment issues - there's so many other things we need to fix (and/or test) as well.

jfoug commented 6 years ago

The one I really want help with is wpapsk That format is a nightmare. I turned off SIMD for BE builds on several formats which are harder than trivial, and some of which may not have a huge impact. I now we have a SAP fellow on team, and he may want to have a shot at it, but those formats also are not trivial.

I am going to make a sibling bug report on these (and close this one out when the other formats are done). That way, we have documentation of WHICH formats we have skipped. Currently I did this for dynamic, and put a bug specifically about that 'format'. I will take a stab at it at some time, but it will NOT be trival. There are simply 100's of locations which will require porting code. I think porting the flat stuff will happen first. BUT once porting starts, I am not are I can turn off SIMD on dyna, but still keep it active for the FLAT formats. It may be all or nothing.

magnumripper commented 6 years ago

Unless WPAPSK already uses the shared PBKDF2-HMAC-SHA1 code, it should. Given that, it really shouldn't be a nightmare.

magnumripper commented 6 years ago

I'm very close to buying a Raspberry Pi or something. Are there any ready-to-use dists using BE for them? I know they're not mainstream... but is there any at all?

jfoug commented 6 years ago

RPI are not BE (I think), They are ARM-LE But they are hella cheap, and you can put a ton of distro's on them right away. I have the normal raspberry, then a Kali disto (hella cool for PEN testing, ;) and a couple other distros. You simply drop them onto a 64 mb micro sd, and bam, the thing boots in a second or 2 and is read to go. Shut it down, pull the SD, stick in a new SD, and boom, another distro! They even have a window RT type thing (I was not able to get it really working). It was the raw OS, without display type support.

You can not go wrong with a RPI3, BUT you will get hooked, very easily. I just burned out a bit after completing the PEN testing. My company was SHOCKED at what someone could do for under $100 and with a device about the size of a pack of smokes. I stuck one in a lower level bathroom (hidden in the ceiling tiles). Later I built a tiny little case, and hooked it inline on an easily accessible printer (they were shocked when I provided 800 printed documents including some very sensitive items, and that was on a 2 day run. Anyone with a 'work shirt', looking like they were working on a printer can easily access many companies and do things like that.

But, you really should get a couple. Also get a tiny screen. I got a 7" for 60 or so. They are hella good little boxes. 64 bit, (but no one has ported a 64 bit OS to them yet, so the 64 bit is unusable). They have NEON SIMD. But do not expect them to be a workstation ;)

kholia commented 6 years ago

I am trying to compile BE testing notes in HACKING.md file.

@magnumripper This might be useful for you.

jfoug commented 6 years ago

Down to the last one!!!! Yea. Doing a full -test=0 (takes a while) to make sure there was nothing I missed.

jfoug commented 6 years ago

This bug is now DONE!. I am going to open 2 new bug reports (one is already open) these will be porting for real, dynamic to BE SIMD. The other is for porting the formats I skipped over to BE SIMD.

All of these formats were simply 'disabled' SIMD when building under a BE machine. They currently DO run on a SIMD machine, but do not use SIMD. The new bugs are simply to enable SIMD for real in these formats.

jfoug commented 6 years ago

Looks like we have some non-SIMD failures now, after these ports.

jfoug commented 6 years ago

xsha512 and zipmonster re-ported back for BE non-SIMD 50a67a23c

jfoug commented 6 years ago

All formats have been reported back to non-SIMD BE (and checked also SIMD BE and SIMD/nonSIMD LE), so closing this bug again.