search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.26k stars 2.1k forks source link

clang build fails on well and bull #1511

Closed frank-dittrich closed 7 years ago

frank-dittrich commented 9 years ago

On super, clang isn't installed.

On well:

frank@well:~$ make -s distclean; ./configure CC=clang --disable-cuda && make -s clean && make -s && ../run/john --test=0 --format=opencl
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking whether to compile using MPI... no
checking for gcc... clang
checking whether the C compiler works... yes
[...]
clang: warning: argument unused during compilation: '-arch host'
configure: creating ./dynamic_big_crypt.c

checking for john.local.conf... exists

Configured for building John the Ripper jumbo:

Target CPU .................................. x86_64 AVX, 64-bit LE
AES-NI support .............................. run-time detection
Target OS ................................... linux-gnu
Cross compiling ............................. no
Legacy arch header .......................... x86-64.h

Optional libraries/features found:
Experimental code ........................... no
OpenMPI support (default disabled) .......... no
Fork support ................................ yes
OpenMP support .............................. no
OpenCL support .............................. yes
CUDA support ................................ no
Generic crypt(3) format ..................... yes
Rexgen (extra cracking mode) ................ no
GMP (PRINCE mode and faster SRP formats) .... yes
PCAP (vncpcap2john and SIPdump) ............. yes
Z (pkzip format, gpg2john) .................. yes
BZ2 (gpg2john extra decompression logic) .... yes
128-bit integer (faster PRINCE mode) ........ yes
Memory map (share/page large files) ......... yes

Development options (these may hurt performance when enabled):
Memdbg memory debugging settings ............ disabled
AddressSanitizer ("ASan") ................... disabled
UndefinedBehaviorSanitizer ("UbSan") ........ disabled

Install missing libraries to get any needed features that were omitted.

Configure finished.  Now 'make clean && make -s' to compile.
clang: warning: argument unused during compilation: '-arch host'
[...]
clang: warning: argument unused during compilation: '-arch host'
In file included from dynamic_big_crypt.c:88:
./gost.h:80:10: warning: 'bswap_32' macro redefined
# define bswap_32(x) _JtR_Swap_32(x)
         ^
/usr/include/byteswap.h:33:9: note: previous definition is here
#define bswap_32(x) __bswap_32 (x)
        ^
In file included from dynamic_big_crypt.c:88:
./gost.h:102:10: warning: 'bswap_64' macro redefined
# define bswap_64(x) _JtR_Swap_64(x)
         ^
/usr/include/byteswap.h:37:10: note: previous definition is here
# define bswap_64(x) __bswap_64 (x)
         ^
2 warnings generated.
clang: warning: argument unused during compilation: '-arch host'
In file included from dynamic_compiler.c:149:
./gost.h:80:10: warning: 'bswap_32' macro redefined
# define bswap_32(x) _JtR_Swap_32(x)
         ^
/usr/include/byteswap.h:33:9: note: previous definition is here
#define bswap_32(x) __bswap_32 (x)
        ^
In file included from dynamic_compiler.c:149:
./gost.h:102:10: warning: 'bswap_64' macro redefined
# define bswap_64(x) _JtR_Swap_64(x)
         ^
/usr/include/byteswap.h:37:10: note: previous definition is here
# define bswap_64(x) __bswap_64 (x)
         ^
2 warnings generated.
clang: warning: argument unused during compilation: '-arch host'
[...]
clang: warning: argument unused during compilation: '-arch host'
fatal error: error in backend: Cannot select: 0x1c998e0: ch = store 0x1c987d0, 0x1c70080, 0x1c988d0, 0x1c1fb50<ST8[%183]> [ID=182] dbg:HDAA_fmt_plug.c:373:17
  0x1c70080: x86mmx = llvm.x86.mmx.padd.d 0x1c224a0, 0x1c74b10, 0x1c6fb80 [ORD=488] [ID=180]
    0x1c224a0: i64 = Constant<607> [ORD=376] [ID=35]
    0x1c74b10: x86mmx = llvm.x86.mmx.padd.d 0x1c224a0, 0x1c74310, 0x1c241c0 [ORD=455] [ID=170]
      0x1c224a0: i64 = Constant<607> [ORD=376] [ID=35]
      0x1c74310: x86mmx = llvm.x86.mmx.punpckhbw 0x1c229a0, 0x1c8a240, 0x1c8a440 [ORD=445] [ID=168]
        0x1c229a0: i64 = Constant<663> [ORD=368] [ID=34]
        0x1c8a240: x86mmx = llvm.x86.mmx.pand 0x1c249c0, 0x1c8a140, 0x1c20550 [ORD=433] [ID=166]
          0x1c249c0: i64 = Constant<615> [ORD=356] [ID=32]
          0x1c8a140: x86mmx = llvm.x86.mmx.psrli.q 0x1c242c0, 0x1c49ad0, 0x1c243c0 [ORD=432] [ID=164]
            0x1c242c0: i64 = Constant<653> [ORD=354] [ID=31]
            0x1c49ad0: x86mmx,ch = load 0x1c89d40, 0x1c8a640, 0x1c1fb50<LD8[%121]> [ID=163]
              0x1c8a640: i64 = or 0x1c268f0, 0x1c89a40 [ID=61] dbg:HDAA_fmt_plug.c:345:20
                0x1c268f0: i64 = FrameIndex<1> [ORD=342] [ID=25]
                0x1c89a40: i64 = Constant<8> [ORD=426] [ID=37]
              0x1c1fb50: i64 = undef [ORD=319] [ID=4]
            0x1c243c0: i32 = Constant<4> [ORD=354] [ID=30]
          0x1c20550: x86mmx,ch = load 0x1c246c0, 0x1c2cf50, 0x1c1fb50<LD8[FixedStack5](align=16)> [ID=81]
            0x1c2cf50: i64 = FrameIndex<5> [ID=46]
            0x1c1fb50: i64 = undef [ORD=319] [ID=4]
        0x1c8a440: x86mmx = llvm.x86.mmx.pand 0x1c249c0, 0x1c49ad0, 0x1c20550 [ORD=435] [ID=165]
          0x1c249c0: i64 = Constant<615> [ORD=356] [ID=32]
          0x1c49ad0: x86mmx,ch = load 0x1c89d40, 0x1c8a640, 0x1c1fb50<LD8[%121]> [ID=163]
            0x1c8a640: i64 = or 0x1c268f0, 0x1c89a40 [ID=61] dbg:HDAA_fmt_plug.c:345:20
              0x1c268f0: i64 = FrameIndex<1> [ORD=342] [ID=25]
              0x1c89a40: i64 = Constant<8> [ORD=426] [ID=37]
            0x1c1fb50: i64 = undef [ORD=319] [ID=4]
          0x1c20550: x86mmx,ch = load 0x1c246c0, 0x1c2cf50, 0x1c1fb50<LD8[FixedStack5](align=16)> [ID=81]
            0x1c2cf50: i64 = FrameIndex<5> [ID=46]
            0x1c1fb50: i64 = undef [ORD=319] [ID=4]
      0x1c241c0: x86mmx = bitcast 0x1c245c0 [ID=76]
        0x1c245c0: i64,ch = load 0x1b8cb58, 0x1c49ed0, 0x1c1fb50<LD8[ConstantPool](align=16)> [ID=70]
          0x1c49ed0: i64 = X86ISD::Wrapper 0x1c4a1d0 [ID=64]
            0x1c4a1d0: i64 = TargetConstantPool<<4 x i32> <i32 101058054, i32 101058054, i32 undef, i32 undef>> 0 [ID=49]
          0x1c1fb50: i64 = undef [ORD=319] [ID=4]
    0x1c6fb80: x86mmx = llvm.x86.mmx.padd.d 0x1c224a0, 0x1c6f680, 0x1c255e0 [ORD=482] [ID=178]
      0x1c224a0: i64 = Constant<607> [ORD=376] [ID=35]
      0x1c6f680: x86mmx = llvm.x86.mmx.pmull.w 0x1c2a630, 0x1c8b250, 0x1c26ff0 [ORD=475] [ID=176]
        0x1c2a630: i64 = Constant<635> [ORD=402] [ID=36]
        0x1c8b250: x86mmx = llvm.x86.mmx.pand 0x1c249c0, 0x1c8b150, 0x1c22aa0 [ORD=465] [ID=174]
          0x1c249c0: i64 = Constant<615> [ORD=356] [ID=32]
          0x1c8b150: x86mmx = llvm.x86.mmx.psrli.q 0x1c242c0, 0x1c74b10, 0x1c243c0 [ORD=464] [ID=172]
            0x1c242c0: i64 = Constant<653> [ORD=354] [ID=31]
            0x1c74b10: x86mmx = llvm.x86.mmx.padd.d 0x1c224a0, 0x1c74310, 0x1c241c0 [ORD=455] [ID=170]
              0x1c224a0: i64 = Constant<607> [ORD=376] [ID=35]
              0x1c74310: x86mmx = llvm.x86.mmx.punpckhbw 0x1c229a0, 0x1c8a240, 0x1c8a440 [ORD=445] [ID=168]
                0x1c229a0: i64 = Constant<663> [ORD=368] [ID=34]
                0x1c8a240: x86mmx = llvm.x86.mmx.pand 0x1c249c0, 0x1c8a140, 0x1c20550 [ORD=433] [ID=166]
                  0x1c249c0: i64 = Constant<615> [ORD=356] [ID=32]
                  0x1c8a140: x86mmx = llvm.x86.mmx.psrli.q 0x1c242c0, 0x1c49ad0, 0x1c243c0 [ORD=432] [ID=164]

                  0x1c20550: x86mmx,ch = load 0x1c246c0, 0x1c2cf50, 0x1c1fb50<LD8[FixedStack5](align=16)> [ID=81]

                0x1c8a440: x86mmx = llvm.x86.mmx.pand 0x1c249c0, 0x1c49ad0, 0x1c20550 [ORD=435] [ID=165]
                  0x1c249c0: i64 = Constant<615> [ORD=356] [ID=32]
                  0x1c49ad0: x86mmx,ch = load 0x1c89d40, 0x1c8a640, 0x1c1fb50<LD8[%121]> [ID=163]

                  0x1c20550: x86mmx,ch = load 0x1c246c0, 0x1c2cf50, 0x1c1fb50<LD8[FixedStack5](align=16)> [ID=81]

              0x1c241c0: x86mmx = bitcast 0x1c245c0 [ID=76]
                0x1c245c0: i64,ch = load 0x1b8cb58, 0x1c49ed0, 0x1c1fb50<LD8[ConstantPool](align=16)> [ID=70]
                  0x1c49ed0: i64 = X86ISD::Wrapper 0x1c4a1d0 [ID=64]

                  0x1c1fb50: i64 = undef [ORD=319] [ID=4]
            0x1c243c0: i32 = Constant<4> [ORD=354] [ID=30]
          0x1c22aa0: x86mmx,ch = load 0x1c20450, 0x1c2dc60, 0x1c1fb50<LD8[FixedStack4](align=16)> [ID=82]
            0x1c2dc60: i64 = FrameIndex<4> [ID=45]
            0x1c1fb50: i64 = undef [ORD=319] [ID=4]
        0x1c26ff0: x86mmx = bitcast 0x1c1ff50 [ID=78]
          0x1c1ff50: i64,ch = load 0x1b8cb58, 0x1c21da0, 0x1c1fb50<LD8[ConstantPool](align=16)> [ID=72]
            0x1c21da0: i64 = X86ISD::Wrapper 0x1c221a0 [ID=66]
              0x1c221a0: i64 = TargetConstantPool<<4 x i32> <i32 2555943, i32 2555943, i32 undef, i32 undef>> 0 [ID=51]
            0x1c1fb50: i64 = undef [ORD=319] [ID=4]
      0x1c255e0: x86mmx = bitcast 0x1c2da60 [ID=79]
        0x1c2da60: i64,ch = load 0x1b8cb58, 0x1c20150, 0x1c1fb50<LD8[ConstantPool](align=16)> [ID=73]
          0x1c20150: i64 = X86ISD::Wrapper 0x1c20350 [ID=67]
            0x1c20350: i64 = TargetConstantPool<<4 x i32> <i32 707406378, i32 707406378, i32 undef, i32 undef>> 0 [ID=52]
          0x1c1fb50: i64 = undef [ORD=319] [ID=4]
  0x1c988d0: i64 = add 0x1c25ce0, 0x1c2d760 [ORD=491] [ID=56] dbg:HDAA_fmt_plug.c:373:17
    0x1c25ce0: i64 = FrameIndex<2> [ORD=347] [ID=27]
    0x1c2d760: i64 = Constant<24> [ORD=318] [ID=2]
  0x1c1fb50: i64 = undef [ORD=319] [ID=4]
make[1]: *** [HDAA_fmt_plug.o] Error 1
make: *** [default] Error 2

May be clang is among the

443 packages can be updated.
280 updates are security updates.

On bull:

909 packages can be updated.
677 updates are security updates.
[...]
frank@bull:~$ clang --version
Ubuntu clang version 3.0-6ubuntu3 (tags/RELEASE_30/final) (based on LLVM 3.0)
Target: x86_64-pc-linux-gnu
Thread model: posix

Trying to build on bull produces the same results as on well.

frank-dittrich commented 9 years ago

I wanted to check whether we can reproduce the Travis OpenCL format errors for the failing clang build. On well, after moving these plugins out of the way 

crc32_fmt_plug.c  gost3411-2012-sse41_plug.c  HDAA_fmt_plug.c  stribog_fmt_plug.c

(because clang has trouble compiling them), I get just one failed OpenCL format self test:

Testing: rar-opencl, RAR3 (length 5) [SHA1 OpenCL AES]... FAILED (valid (before init))

(already known, see issue https://github.com/magnumripper/JohnTheRipper/issues/1499) and these warnings:

Testing: strip-opencl, STRIP Password Manager [PBKDF2-SHA1 OpenCL]... Warning: salt() returned misaligned pointer
Testing: sxc-opencl, StarOffice .sxc [PBKDF2-SHA1 OpenCL Blowfish]... Warning: salt() returned misaligned pointer

And I get these failing CPU formats:

$ ./john --test=0 --format=cpu | grep -v PASS
Testing: descrypt, traditional crypt(3) [DES 128/128 AVX-16]... FAILED (cmp_all(2))

Testing: bsdicrypt, BSDI crypt(3) ("_J9..", 725 iterations) [DES 128/128 AVX-16]... FAILED (cmp_all(2))

Testing: LM [DES 128/128 AVX-16]... FAILED (cmp_all(2))

Testing: mschapv2-naive, MSCHAPv2 C/R [MD4 DES DES 128/128 AVX-16 naive]... FAILED (cmp_all(2))
-
Testing: netntlm-naive, NTLMv1 C/R [MD4 DES (ESS MD5) DES 128/128 AVX-16 naive]... FAILED (cmp_all(2))

5 out of 311 tests have FAILED

$ ./john --list=build-info Version: 1.8.0.6-jumbo-1-252-g2c9182e+ Build: linux-gnu 64-bit AVX-ac SIMD: AVX, interleaving: MD4:4 MD5:5 SHA1:2 SHA256:1 SHA512:1 $JOHN is ./ Format interface version: 13 Max. number of reported tunable costs: 3 Rec file version: REC4 Charset file version: CHR3 CHARSET_MIN: 1 (0x01) CHARSET_MAX: 255 (0xff) CHARSET_LENGTH: 24 Max. Markov mode level: 400 Max. Markov mode password length: 30 clang version: 3.0 (tags/RELEASE_30/final) (gcc 4.2.1 compatibility) GNU libc version: 2.15 (loaded: 2.15) OpenCL library version: 1.2 Crypto library: OpenSSL OpenSSL library version: 01000100f OpenSSL 1.0.1 14 Mar 2012 GMP library version: 5.0.2 File locking: fcntl() fseek(): fseek ftell(): ftell fopen(): fopen memmem(): System's

frank-dittrich commented 9 years ago

BTW: In addition to the tons of

clang: warning: argument unused during compilation: '-arch host'

warnings, I also get these warnings:

In file included from dynamic_big_crypt.c:88:
./gost.h:80:10: warning: 'bswap_32' macro redefined
# define bswap_32(x) _JtR_Swap_32(x)
         ^
/usr/include/byteswap.h:33:9: note: previous definition is here
#define bswap_32(x) __bswap_32 (x)
        ^
In file included from dynamic_big_crypt.c:88:
./gost.h:102:10: warning: 'bswap_64' macro redefined
# define bswap_64(x) _JtR_Swap_64(x)
         ^
/usr/include/byteswap.h:37:10: note: previous definition is here
# define bswap_64(x) __bswap_64 (x)
         ^
2 warnings generated.
clang: warning: argument unused during compilation: '-arch host'
In file included from dynamic_compiler.c:149:
./gost.h:80:10: warning: 'bswap_32' macro redefined
# define bswap_32(x) _JtR_Swap_32(x)
         ^
/usr/include/byteswap.h:33:9: note: previous definition is here
#define bswap_32(x) __bswap_32 (x)
        ^
In file included from dynamic_compiler.c:149:
./gost.h:102:10: warning: 'bswap_64' macro redefined
# define bswap_64(x) _JtR_Swap_64(x)
         ^
/usr/include/byteswap.h:37:10: note: previous definition is here
# define bswap_64(x) __bswap_64 (x)
         ^
2 warnings generated.
cracker.c:388:3: warning: data argument not used by format string [-Wformat-extra-args]
                crk_db->salt_count);
                ^
1 warning generated.
john.c:732:3: warning: data argument not used by format string [-Wformat-extra-args]
                database.salt_count);
                ^
1 warning generated.
gpg2john.c:1889:3: warning: if statement has empty body [-Wempty-body]
                ;
                ^
gpg2john.c:1892:3: warning: if statement has empty body [-Wempty-body]
                ;
                ^
2 warnings generated.
frank-dittrich commented 9 years ago

The

fatal error: error in backend: Cannot select: 0x30e0460: ch = store 0x30df350, 0x30d6790, 0x30df450, 0x3066ec0<ST8[%183]> [ID=182] dbg:HDAA_fmt_plug.c:373:17

message has already been mentioned above. These are the other error messages:

fatal error: error in backend: Cannot select: intrinsic %llvm.x86.sse42.crc32.32.8
make[1]: *** [crc32_fmt_plug.o] Error 1
fatal error: error in backend: Cannot select: intrinsic %llvm.x86.mmx.emms
make[1]: *** [gost3411-2012-sse41_plug.o] Error 1

and after moving gost3411-2012-sse41_plug.c out of the way:

stribog_fmt_plug.o: In function `stribog256_init':
/space/home/frank/git/JtR/src/stribog_fmt_plug.c:247: undefined reference to `GOST34112012Init'
stribog_fmt_plug.o: In function `stribog_final':
/space/home/frank/git/JtR/src/stribog_fmt_plug.c:279: undefined reference to `GOST34112012Final'
stribog_fmt_plug.o: In function `stribog512_init':
/space/home/frank/git/JtR/src/stribog_fmt_plug.c:254: undefined reference to `GOST34112012Init'
stribog_fmt_plug.o: In function `stribog_final':
/space/home/frank/git/JtR/src/stribog_fmt_plug.c:279: undefined reference to `GOST34112012Final'
stribog_fmt_plug.o: In function `stribog_update':
/space/home/frank/git/JtR/src/stribog_fmt_plug.c:270: undefined reference to `GOST34112012Update'
/space/home/frank/git/JtR/src/stribog_fmt_plug.c:271: undefined reference to `GOST34112012Update'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [../run/john] Error 1
make: *** [default] Error 2
magnumripper commented 9 years ago

I opened #1512 for the warnings specific to recent dynamic changes

magnumripper commented 9 years ago

I wanted to check whether we can reproduce the Travis OpenCL format errors for the failing clang build

As far as I understand those problems has nothing to do with clang. It's just that gcc segfaults earlier so we don't see them.

frank-dittrich commented 9 years ago

@magnumripper clang not being to blame for the OpenCL format errors is probably correct, but I didn't want to jump to conclusions I cannot prove.

magnumripper commented 9 years ago

I'd even say it's the same bug(s) (in shared or duplicated code) in most of them.

frank-dittrich commented 8 years ago

I wanted to test whether these problems still exist, but apparently, there's no clang on well and super anymore, and bull is refusing ssh connections.

Not sure what to do now. Just close this issue? Ask Solar to install clang, so that we can test clang builds?

[frank@super src]$ make -s distclean; ./configure CC=clang --disable-cuda && make -s clean && make -s && ../run/john --test=0 --format=opencl
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking whether to compile using MPI... no
checking for gcc... clang
checking whether the C compiler works... no
configure: error: in `/home/frank/git/JtR/src':
configure: error: C compiler cannot create executables
See `config.log' for more details
[frank@super src]$ which clang
/usr/bin/which: no clang in (...)
magnumripper commented 8 years ago

Yes I think clang would be a good thing on well & super (different versions of it would be even better).

claudioandre-br commented 7 years ago

The problem is gone:

claudio@well:~/bleeding/src$ make -s distclean; ./configure CC=clang && make -s clean && make -s && ../run/john --test=0 --format=opencl
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking whether to compile using MPI... no
checking for gcc... clang
[...]
Device 0: Bonaire [AMD Radeon HD 7700 Series]
[...]
Testing: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... PASS
$ ../run/john --list=build-info
Version: 1.8.0-jumbo-1-5393-gc19f75f
Build: linux-gnu 64-bit AVX2-ac
SIMD: AVX2, interleaving: MD4:4 MD5:5 SHA1:2 SHA256:1 SHA512:1
$JOHN is ../run/
Format interface version: 14
Max. number of reported tunable costs: 3
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
Max. Markov mode level: 400
Max. Markov mode password length: 30
clang version: 3.4.2 (tags/RELEASE_34/dot2-final) (gcc 4.2.1 compatibility)
GNU libc version: 2.17 (loaded: 2.17)
OpenCL headers version: 2.0
Crypto library: OpenSSL
OpenSSL library version: 01000105f
OpenSSL 1.0.1e-fips 11 Feb 2013
GMP library version: 6.0.0
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
solardiz commented 7 years ago

The problem is gone:

Device 0: Bonaire [AMD Radeon HD 7700 Series]

Claudio, FWIW you're running these tests on a GPU that is under heavy concurrent use, with much of its memory allocated by the unrelated job. Maybe this somehow affects which tests pass or fail (at least it might if any uninitialized memory is involved, or for the case of new failures in case together these jobs exhaust a shared resource).

claudioandre-br commented 7 years ago

Well, since we were seeing a clean test with clang 3.0 on Travis (until recently). Now, a clean test using 3.4.