Updated rexgen library from git does not define c_simplestring_bufferaddress

[bill-e-ghote]

Rexgen page on code.google.com states: This project is currently moving to https://github.com/teeshop/rexgen

Installed rexgen from git source tree. JtR bleeding-jumbo from git source tree:

Configured for building John the Ripper 1.8.0.2-bleeding-jumbo:

Target CPU .................................. x86_64 AVX, 64-bit LE
AES-NI support .............................. run-time detection
Target OS ................................... linux-gnu
Cross compiling ............................. no
Legacy arch header .......................... x86-64.h
OpenMPI support (default disabled) .......... no
Fork support ................................ yes
OpenMP support .............................. yes
OpenCL support .............................. yes
CUDA support ................................ no
Generic crypt(3) mode ....................... yes

Optional libraries found:
Rexgen (extra cracking mode) ................ yes
GMP (performance) ........................... yes
NSS/NSPR (Mozilla format) ................... yes
Kerberos5 (krb5-18/23 formats) .............. yes (MIT)
PCAP (vncpcap2john and SIPdump) ............. yes
BZ2 (gpg2john extra decompression logic) .... yes

Compilation continues normally until final link step:

gcc jumbo.o john-mpi.o DES_fmt.o DES_std.o DES_bs.o DES_bs_b.o BSDI_fmt.o MD5_fmt.o MD5_std.o BF_fmt.o BF_std.o scrypt_fmt.o escrypt/crypto_scrypt-best.o escrypt/crypto_scrypt-common.o escrypt/sha256.o AFS_fmt.o LM_fmt.o trip_fmt.o timer.o md5_go.o md5.o rc4.o hmacmd5.o base64.o md4.o sha2.o dynamic_fmt.o dynamic_parser.o dynamic_preloads.o dynamic_utils.o dynamic_big_crypt.o ssh2john.o pfx2john.o unrarcmd.o unrarfilter.o unrarhlp.o unrar.o unrarppm.o unrarvm.o rar2john.o zip2john.o gladman_hmac.o gladman_pwd2key.o racf2john.o keychain2john.o kwallet2john.o pwsafe2john.o dmg2john.o keepass2john.o keyring2john.o keystore2john.o truecrypt_volume2john.o hccap2john.o mozilla2john.o 7z_fmt_plug.o BFEgg_fmt_plug.o DMD5_fmt_plug.o DOMINOSEC_fmt_plug.o EPI_fmt_plug.o FGT_fmt_plug.o HDAA_fmt_plug.o IPB2_fmt_plug.o KRB4_fmt_plug.o KRB4_std_plug.o KRB5_fmt_plug.o KRB5_std_plug.o KeccakDuplex_plug.o KeccakF-1600-opt32_plug.o KeccakF-1600-opt64_plug.o KeccakNISTInterface_plug.o KeccakSponge_plug.o KeyDBCracker_plug.o MSCHAPv2_bs_fmt_plug.o NETLM_fmt_plug.o NETLMv2_fmt_plug.o NETNTLM_bs_fmt_plug.o NETNTLMv2_fmt_plug.o NETSPLITLM_fmt_plug.o NS_fmt_plug.o NT_fmt_plug.o PHPS_fmt_plug.o PO_fmt_plug.o SKEY_fmt_plug.o SKEY_jtr_plug.o SybaseASE_fmt_plug.o SybasePROP_fmt_plug.o XSHA512_fmt_plug.o XSHA_fmt_plug.o agilekeychain_fmt_plug.o aix_smd5_fmt_plug.o aix_ssha_fmt_plug.o androidfde_fmt_plug.o asaMD5_fmt_plug.o bitcoin_fmt_plug.o blackberry_ES10_fmt_plug.o blake2b-ref_plug.o blake2b_plug.o blockchain_fmt_plug.o chap_fmt_plug.o citrix_ns_fmt_plug.o clipperz_srp_fmt_plug.o cloudkeychain_fmt_plug.o common_opencl_pbkdf2_plug.o crc32_fmt_plug.o crypt-sha1_fmt_plug.o crypt-sha1_plug.o cryptsha256_fmt_plug.o cryptsha512_fmt_plug.o cuda_cryptmd5_fmt_plug.o cuda_cryptsha256_fmt_plug.o cuda_cryptsha512_fmt_plug.o cuda_mscash2_fmt_plug.o cuda_mscash_fmt_plug.o cuda_phpass_fmt_plug.o cuda_pwsafe_fmt_plug.o cuda_rawsha512_fmt_plug.o cuda_wpapsk_fmt_plug.o cuda_xsha512_fmt_plug.o django_fmt_plug.o django_scrypt_fmt_plug.o dmg_fmt_plug.o dragonfly3_fmt_plug.o dragonfly4_fmt_plug.o drupal7_fmt_plug.o ecryptfs_fmt_plug.o efs_fmt_plug.o encfs_fmt_plug.o episerver_fmt_plug.o feal8_plug.o formspring_fmt_plug.o gost_fmt_plug.o gpg_fmt_plug.o haval_fmt_plug.o haval_plug.o hmacMD5_fmt_plug.o hmacSHA1_fmt_plug.o hmacSHA224_fmt_plug.o hmacSHA256_fmt_plug.o hmacSHA384_fmt_plug.o hmacSHA512_fmt_plug.o hmailserver_fmt_plug.o idea_plug.o ike_fmt_plug.o keepass_fmt_plug.o keychain_fmt_plug.o keyring_fmt_plug.o keystore_fmt_plug.o krb5-18_fmt_plug.o krb5-23_fmt_plug.o krb5pa-md5_fmt_plug.o krb5pa-sha1_fmt_plug.o kwallet_fmt_plug.o lastpass_fmt_plug.o lastpass_sniffed_fmt_plug.o lotus5_fmt_plug.o lotus85_fmt_plug.o lowpbe_plug.o luks_fmt_plug.o md2_fmt_plug.o md2_plug.o md4_gen_fmt_plug.o mediawiki_fmt_plug.o mongodb_fmt_plug.o mozilla_des_plug.o mozilla_fmt_plug.o mscash1_fmt_plug.o mscash2_fmt_plug.o mssql-old_fmt_plug.o mssql05_fmt_plug.o mssql12_fmt_plug.o mysqlSHA1_fmt_plug.o mysql_fmt_plug.o mysql_netauth_fmt_plug.o net_md5_fmt_plug.o net_sha1_fmt_plug.o nsldap_fmt_plug.o nt2_fmt_plug.o ntlmv1_mschapv2_fmt_plug.o nukedclan_fmt_plug.o o5logon_fmt_plug.o odf_fmt_plug.o office_fmt_plug.o oldoffice_fmt_plug.o opencl_DES_bs_b_plug.o opencl_DES_bs_plug.o opencl_DES_fmt_plug.o opencl_agilekeychain_fmt_plug.o opencl_bf_fmt_plug.o opencl_bf_std_plug.o opencl_blockchain_fmt_plug.o opencl_cryptmd5_fmt_plug.o opencl_cryptsha256_fmt_plug.o opencl_cryptsha512_fmt_plug.o opencl_dmg_fmt_plug.o opencl_encfs_fmt_plug.o opencl_gpg_fmt_plug.o opencl_keychain_fmt_plug.o opencl_keyring_fmt_plug.o opencl_krb5pa-md5_fmt_plug.o opencl_krb5pa-sha1_fmt_plug.o opencl_lotus5_fmt_plug.o opencl_mscash2_fmt_plug.o opencl_mysqlsha1_fmt_plug.o opencl_nsldaps_fmt_plug.o opencl_nt_fmt_plug.o opencl_ntlmv2_fmt_plug.o opencl_o5logon_fmt_plug.o opencl_odf_aes_fmt_plug.o opencl_odf_fmt_plug.o opencl_office2007_fmt_plug.o opencl_office2010_fmt_plug.o opencl_office2013_fmt_plug.o opencl_pbkdf2_hmac_sha256_fmt_plug.o opencl_pbkdf2_hmac_sha512_fmt_plug.o opencl_phpass_fmt_plug.o opencl_pwsafe_fmt_plug.o opencl_rakp_fmt_plug.o opencl_rar_fmt_plug.o opencl_rawmd4_fmt_plug.o opencl_rawmd5_fmt_plug.o opencl_rawsha1_fmt_plug.o opencl_rawsha256_fmt_plug.o opencl_rawsha512-ng_fmt_plug.o opencl_rawsha512_fmt_plug.o opencl_sha2_common_plug.o opencl_strip_fmt_plug.o opencl_sxc_fmt_plug.o opencl_wpapsk_fmt_plug.o opencl_xsha512_fmt_plug.o opencl_zip_fmt_plug.o openssl_enc_fmt_plug.o oracle11_fmt_plug.o oracle_fmt_plug.o osc_fmt_plug.o panama_fmt_plug.o panama_plug.o pbkdf2-hmac-sha1_fmt_plug.o pbkdf2-hmac-sha512_fmt_plug.o pbkdf2_hmac_sha256_fmt_plug.o pdf_fmt_plug.o pdfcrack_md5_plug.o pfx_fmt_plug.o phpassMD5_fmt_plug.o pixMD5_fmt_plug.o pkzip_fmt_plug.o postgres_fmt_plug.o pst_fmt_plug.o putty_fmt_plug.o pwsafe_fmt_plug.o racf_fmt_plug.o radmin_fmt_plug.o rakp_fmt_plug.o rar5_fmt_plug.o rar_fmt_plug.o rawBLAKE2_512_fmt_plug.o rawKeccak_256_fmt_plug.o rawKeccak_512_fmt_plug.o rawMD4_fmt_plug.o rawMD5_fmt_plug.o rawSHA0_fmt_plug.o rawSHA1_fmt_plug.o rawSHA1_linkedIn_fmt_plug.o rawSHA1_ng_fmt_plug.o rawSHA224_fmt_plug.o rawSHA256_fmt_plug.o rawSHA256_ng_fmt_plug.o rawSHA256_ng_i_fmt_plug.o rawSHA384_fmt_plug.o rawSHA512_fmt_plug.o rawSHA512_ng_fmt_plug.o rawSHA512_ng_i_fmt_plug.o rawmd5u_fmt_plug.o ripemd_fmt_plug.o ripemd_plug.o salted_sha1_fmt_plug.o sapB_fmt_plug.o sapG_fmt_plug.o sha1_gen_fmt_plug.o siemens-s7_fmt_plug.o sip_fmt_plug.o skein_fmt_plug.o skein_plug.o snefru_fmt_plug.o snefru_plug.o ssh_fmt_plug.o ssh_ng_fmt_plug.o ssha512_fmt_plug.o strip_fmt_plug.o sunmd5_fmt_plug.o sxc_fmt_plug.o syb-prop_repro_plug.o tcp_md5_fmt_plug.o tiger_fmt_plug.o tiger_plug.o truecrypt_fmt_plug.o uaf_encode_plug.o vms_fmt_plug.o vnc_fmt_plug.o wbb3_fmt_plug.o whirlpool_fmt_plug.o whirlpool_plug.o wow_srp_fmt_plug.o wpapsk_fmt_plug.o zip_fmt_plug.o plugin.o dummy.o gost.o common-gpu.o batch.o bench.o charset.o common.o compiler.o config.o cracker.o crc32.o external.o formats.o getopt.o idle.o inc.o john.o list.o loader.o logger.o mask.o math.o memory.o misc.o options.o params.o path.o recovery.o rpp.o rules.o signals.o single.o status.o tty.o wordlist.o mkv.o mkvlib.o listconf.o fake_salts.o win32_memmap.o unicode.o unshadow.o unafs.o undrop.o regex.o c3_fmt.o unique.o putty2john.o gpg2john.o memdbg.o common-opencl.o opencl_autotune.o x86-64.o sse-intrinsics.o -g  -L/usr/local/lib   -lssl -lcrypto   -fopenmp -lgmp -lskey -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4  -lrexgen  -lk5crypto -lkrb5   -lOpenCL  -lm -pthread  -lm -lz -ldl -lcrypt  -lbz2 aes/aes.a -o ../run/john
regex.o: In function `do_regex_crack_as_rules':
/home/wdawson/Software/bleeding-jumbo/src/regex.c:163: undefined reference to `c_simplestring_terminate'
/home/wdawson/Software/bleeding-jumbo/src/regex.c:164: undefined reference to `c_simplestring_bufferaddress'
regex.o: In function `do_regex_crack':
/home/wdawson/Software/bleeding-jumbo/src/regex.c:200: undefined reference to `c_simplestring_terminate'
/home/wdawson/Software/bleeding-jumbo/src/regex.c:201: undefined reference to `c_simplestring_bufferaddress'
collect2: error: ld returned 1 exit status
make[1]: *** [../run/john] Error 1
make[1]: Leaving directory `/home/wdawson/Software/bleeding-jumbo/src'
make: *** [default] Error 2

Comparison of rexgen code from git vs. previous code pulled from Google svn tree reveal - no reference to c_simplestring_bufferaddress in new code base.

[magnumripper]

Thanks for reporting this. Hopefully @jfoug will have a look at it.

[jfoug]

For now, I have sort of given up on rexgen. I can not build it any more as the recent changes (which do nothing for the lib) require a higher cmake version than I have.

[magnumripper]

More details in #693 which was closed as dupe.

[magnumripper]

@jfoug I can build latest rexgen with cmake 2.8.12.2, I thought you were using the same version?

[magnumripper]

Unfortunately teeshop/rexgen@c88601f is still the latest version we can use, and nothing has happened in that repo since June 18.

[magnumripper]

835ab0f updates JtR to work with latest code from https://github.com/teeshop/rexgen. Older versions are no longer supported (and ./configure will not enable regex mode if you have an old one).

[magnumripper]

BTW rexgen currently has to be cloned like this

git clone --recursive https://github.com/teeshop/rexgen.git

...and it requires icu4c lib and headers.

[frank-dittrich]

I still cannot build. Reason: The rexgen header files used to be installed to /usr/local/include/librexgen/api/c/, but now are installed to /usr/local/include/<librexgen/c, but john's regex.h still has

#include <librexgen/api/c/librexgen.h>

instead of

#include <librexgen/c/librexgen.h>

The configure script correctly checks for librexgen/c/librexgen.h. I wonder how it forked for you without the regex.h change.

May be discuss with Jan whether he intends to keep this API stable or whether we should expect future changes.

[magnumripper]

That include path may something he missed while patching (I guess we should commit it), and maybe he had an older file in the old loc. I have yet to succeed in building rexgen library on OSX due to other problems. We'll sort it out sooner or later.

[magnumripper]

May be discuss with Jan whether he intends to keep this API stable or whether we should expect future changes.

He does plan another change, for performance reasons (and he'll also get rid of the current dependency of icu4c). He will provide patches. But re-reading what he wrote I think he actually wont break the API this time, we just need the update to get the better performance available.