Is bt.c not compatible with Cygwin?

[magnumripper]

It might be that bt.c uses alarm() in a way that Cygwin can't handle as-is. See http://www.openwall.com/lists/john-users/2018/03/31/1

[solardiz]

Probably unrelated to the issue reported here, but nevertheless:

The if (num_iter == limit) code path in create_tables() will return without a preceding alarm(0); - that call should be added.

The save/restore logic in create_perfect_hash_table() assumes that ITIMER_REAL refers to the same timer that we clobber with the alarm() calls. I don't know whether this is guaranteed or not. Maybe the alarm() calls should be changed to setitimer() ones, so that we explicitly deal with one and the same timer.

[solardiz]

I don't know whether this is guaranteed or not.

It isn't guaranteed:

POSIX: "The interaction between setitimer() and any of alarm(), sleep(), or usleep() is unspecified."

[solardiz]

I think we should change bt.c not to use signals and timers at all. The only reason it does so now is to set the signal_stop variable, which it checks in one place. It should instead record the start time where it now does alarm(3); and check the current time against the start time where it now checks the variable. If this code is reached with JtR's usual signal handling & timer already initialized, then we should simply use status_get_time() in it, so that we only depend on one type of timer working correctly.

[solardiz]

Once we make changes to this code, we should re-test with the 320M password hashes on super (which barely fits in Titan X's memory, and stress-tests the hash table size expansions).

While at it, we should limit printing of the "This many number of hashes have never been tested before [...]" message to a number of hashes higher than that 320M figure, because we did test that many now. We should also fix the grammar. Or remove this message, so that we don't need to keep bumping this figure later.

[magnumripper]

Could it be this simple? Didn't test it yet. Where is that 320M test file located?

diff --git a/src/bt.c b/src/bt.c
index cc95d131d..755d277b2 100644
--- a/src/bt.c
+++ b/src/bt.c
@@ -12,9 +12,8 @@
 #include <stdlib.h>
 #include <string.h>
 #include <sys/time.h>
-#include <signal.h>
-#include <unistd.h>

+#include "status.h"
 #include "misc.h" // error()
 #include "bt_twister.h"
 #include "bt_hash_types.h"
@@ -60,16 +59,8 @@ static auxilliary_offset_data *offset_data;

 unsigned long long total_memory_in_bytes;

-static volatile sig_atomic_t signal_stop;
-
 static unsigned int verbosity;

-static void alarm_handler(int sig)
-{
-   if (sig == SIGALRM)
-       signal_stop = 1;
-}
-
 static unsigned int coprime_check(unsigned int m,unsigned int n)
 {
    unsigned int rem;
@@ -380,6 +371,7 @@ static unsigned int create_tables()
    while (offset_data[i].collisions > 1) {
        OFFSET_TABLE_WORD offset;
        unsigned int num_iter;
+       unsigned int start_time;

        done += offset_data[i].collisions;

@@ -393,7 +385,7 @@ static unsigned int create_tables()
            backtracking = 0;
        }
 #endif
-       alarm(3);
+       start_time = status_get_time();

        num_iter = 0;
        while (!check_n_insert_into_hash_table((unsigned int)offset, &offset_data[i], hash_table_idxs, store_hash_modulo_table_sz) && num_iter < limit) {
@@ -410,12 +402,9 @@ static unsigned int create_tables()
                fprintf(stdout, "\rProgress:%Lf %%, Number of collisions:%u", done / (long double)num_loaded_hashes * 100.00, offset_data[i].collisions);
                fflush(stdout);
            }
-           alarm(0);
        }

-       if (signal_stop) {
-           alarm(0);
-           signal_stop = 0;
+       if (status_get_time() >= start_time + 3) {
            fprintf(stderr, "\nProgress is too slow!! trying next table size.\n");
            bt_free((void **)&hash_table_idxs);
            bt_free((void **)&store_hash_modulo_table_sz);
@@ -461,8 +450,6 @@ static unsigned int create_tables()
        i++;
    }

-   alarm(0);
-
    hash_table_idx = 0;
    while (i < offset_table_size && offset_data[i].collisions > 0) {
        done++;
@@ -558,8 +545,6 @@ unsigned int create_perfect_hash_table(int htype, void *loaded_hashes_ptr,
 {
    long double multiplier_ht, multiplier_ot, inc_ht, inc_ot;
    unsigned int approx_hash_table_sz, approx_offset_table_sz, i, dupe_remove_ht_sz;
-   struct sigaction new_action, old_action;
-   struct itimerval old_it;

    total_memory_in_bytes = 0;

@@ -612,19 +597,6 @@ unsigned int create_perfect_hash_table(int htype, void *loaded_hashes_ptr,
            fprintf(stdout, "Using Hash type 192.\n");
    }

-   new_action.sa_handler = alarm_handler;
-   sigemptyset(&new_action.sa_mask);
-   new_action.sa_flags = 0;
-
-   if (sigaction(SIGALRM, NULL, &old_action) < 0)
-       bt_error("Error retriving signal info.");
-
-   if (sigaction(SIGALRM, &new_action, NULL) < 0)
-       bt_error("Error setting new signal handler.");
-
-   if (getitimer(ITIMER_REAL, &old_it) < 0)
-       bt_error("Error retriving timer info.");
-
    inc_ht = 0.005;
    inc_ot = 0.05;

@@ -731,12 +703,6 @@ unsigned int create_perfect_hash_table(int htype, void *loaded_hashes_ptr,
    *hash_table_sz_ptr = hash_table_size;
    *offset_table_sz_ptr = offset_table_size;

-   if (sigaction(SIGALRM, &old_action, NULL) < 0)
-       bt_error("Error restoring previous signal handler.");
-
-   if (setitimer(ITIMER_REAL, &old_it, NULL) < 0)
-       bt_error("Error restoring previous timer.");
-
    if (!test_tables(num_loaded_hashes, offset_table, offset_table_size, shift64_ot_sz, shift128_ot_sz, verbosity))
        return 0;

[solardiz]

That patch looks mostly good to me. Maybe drop #include <sys/time.h>, and the check should be >=, not >, to closer match the previous behavior. Thanks!

[timajus]

@magnum: i've found the Alarm clock is raised by raise(SIGALRM); in signal.c into sig_timer_emu_tick function

[solardiz]

Thanks, @timajus. I think we need something like @magnumripper's patch above for simplicity and to avoid whatever incompatibility with the timer emulation code we might be triggering.

Separately, we might also reconsider use of the timer emulation code on Windows, which I guess is no longer needed for modern versions of Windows and Cygwin - but that would require separate testing, and it's part of the core tree (unlike bt.c, which is a jumbo addition).

[timajus]

@magnumripper @solardiz Thanks. I've tried the patch posted but still the same Alarm clock..here is the verbosity run: (just added this line

fprintf(stdout,"\nCURRENT %lu LAST %lu timer_emu_interval %lu timer_emu_max %u \n", current, last, timer_emu_interval, timer_emu_max); to signals.c to see that the code does not return in any of the two preceding if and goes to the raise(SIGALRM); Maybe it can be useful to understand the problem...

$ ../run/john.exe --format=descrypt-opencl ../run/hashes.txt --verbosity=5
initUnicode(UNICODE, UTF-8/ISO-8859-1)
UTF-8 -> UTF-8 -> UTF-8
Device 2: GeForce GTX 1050 Ti
Using default input encoding: UTF-8
Loaded 627482 password hashes with 256 different salts (descrypt-opencl, traditional crypt(3) [DES OpenCL])
Remaining 625821 password hashes with 256 different salts
Loaded 6 hashes with 4 different salts to test db from test vectors
Options used: -I ../run/kernels -cl-mad-enable -DSM_MAJOR=6 -DSM_MINOR=1 -cl-nv-verbose -D__GPU__ -DDEVICE_INFO=524306 -DSIZEOF_SIZE_T=8 -DDEV_VER_MAJOR=391 -DDEV_VER_MINOR=35 -D_OPENCL_COMPILER  $JOHN/kernels/DES_bs_hash_checking_kernel.cl
Options used: -I ../run/kernels -cl-mad-enable -DSM_MAJOR=6 -DSM_MINOR=1 -cl-nv-verbose -D__GPU__ -DDEVICE_INFO=524306 -DSIZEOF_SIZE_T=8 -DDEV_VER_MAJOR=391 -DDEV_VER_MINOR=35 -D_OPENCL_COMPILER -D ITER_COUNT=1 -D MASK_ENABLED=0 -D LOC_0=-1 -D LOC_1=-1 -D LOC_2=-1 -D LOC_3=-1 -D IS_STATIC_GPU_MASK=0 -D CONST_CACHE_SIZE=65536 $JOHN/kernels/DES_bs_finalize_keys_kernel.cl
Options used: -I ../run/kernels -cl-mad-enable -DSM_MAJOR=6 -DSM_MINOR=1 -cl-nv-verbose -D__GPU__ -DDEVICE_INFO=524306 -DSIZEOF_SIZE_T=8 -DDEV_VER_MAJOR=391 -DDEV_VER_MINOR=35 -D_OPENCL_COMPILER -D WORK_GROUP_SIZE=0 -D index0=31 -D index1=16 -D index2=17 -D index3=18 -D index4=3 -D index5=4 -D index6=3 -D index7=20 -D index8=21 -D index9=22 -D index10=7 -D index11=8 -D index24=15 -D index25=0 -D index26=1 -D index27=2 -D index28=19 -D index29=20 -D index30=19 -D index31=4 -D index32=5 -D index33=6 -D index34=23 -D index35=24 -D index48=63 -D index49=48 -D index50=49 -D index51=50 -D index52=35 -D index53=36 -D index54=35 -D index55=52 -D index56=53 -D index57=54 -D index58=39 -D index59=40 -D index72=47 -D index73=32 -D index74=33 -D index75=34 -D index76=51 -D index77=52 -D index78=51 -D index79=36 -D index80=37 -D index81=38 -D index82=55 -D index83=56  $JOHN/kernels/DES_bs_kernel_h.cl
Salt compiled from Source:910
GWS: 1048576, LWS: 128
Salt compiled from Binary:910
Salt compiled from Binary:2275
Salt compiled from Binary:990
Salt compiled from Binary:0
Updated internal tables and buffers for salt 910.
Updated internal tables and buffers for salt 2275.
Updated internal tables and buffers for salt 990.
Updated internal tables and buffers for salt 0.
Note: This format may be a lot faster with --mask acceleration (see doc/MASK).
Options used: -I ../run/kernels -cl-mad-enable -DSM_MAJOR=6 -DSM_MINOR=1 -cl-nv-verbose -D__GPU__ -DDEVICE_INFO=524306 -DSIZEOF_SIZE_T=8 -DDEV_VER_MAJOR=391 -DDEV_VER_MINOR=35 -D_OPENCL_COMPILER  $JOHN/kernels/DES_bs_hash_checking_kernel.cl
Options used: -I ../run/kernels -cl-mad-enable -DSM_MAJOR=6 -DSM_MINOR=1 -cl-nv-verbose -D__GPU__ -DDEVICE_INFO=524306 -DSIZEOF_SIZE_T=8 -DDEV_VER_MAJOR=391 -DDEV_VER_MINOR=35 -D_OPENCL_COMPILER -D ITER_COUNT=1 -D MASK_ENABLED=0 -D LOC_0=-1 -D LOC_1=-1 -D LOC_2=-1 -D LOC_3=-1 -D IS_STATIC_GPU_MASK=0 -D CONST_CACHE_SIZE=65536 $JOHN/kernels/DES_bs_finalize_keys_kernel.cl
Note: building per-salt kernels. This takes e.g. 2 hours for 4096 salts.
Salt compiled from Binary:258

CURRENT 101666240 LAST 0 timer_emu_interval 0 timer_emu_max 0
Salt compiled from Binary:514

CURRENT 101666241 LAST 101666240 timer_emu_interval 0 timer_emu_max 0
Alarm clock

[magnumripper]

@solardiz the #include <sys/time.h> is still needed because Sayantan gets random seed from gettimeofday().

Apparently @timajus problem is caused by something else. We should commit the patch anyway but I can't really tell what's the problem there.

Perhaps try this for a start:

diff --git a/src/os-autoconf.h b/src/os-autoconf.h
index 3e1d967dc..d5cf3e491 100644
--- a/src/os-autoconf.h
+++ b/src/os-autoconf.h
@@ -30,7 +30,7 @@

 #ifdef NEED_OS_TIMER

-#if defined(__CYGWIN32__) || defined(__BEOS__) || defined(__MINGW32__) || defined(_MSC_VER) /* || (defined(AMDAPPSDK) && defined(HAVE_OPENCL)) */
+#if 0 //defined(__CYGWIN32__) || defined(__BEOS__) || defined(__MINGW32__) || defined(_MSC_VER) /* || (defined(AMDAPPSDK) && defined(HAVE_OPENCL)) */
 #define OS_TIMER                       0
 #else
 #ifndef _XOPEN_SOURCE

[magnumripper]

I'm now trying the opposite - I'm building with OS_TIMER hard set to 0 and see what happens on my macbook.

[magnumripper]

I see no immediate problem with running emulated timer. @timajus that zippyshare link you posted for your files just tries to feed me malware. Can you please upload them to some better place? Perhaps you can simply attach them to a post right here (unless they're too large for that - I don't know the limits).

[timajus]

made the patch, a couple of warnings when compiling... when running it starts running but after a while alarm clock again: (attached test.zip file) Thanks!

$ make clean -s && make -sj4
signals.c:24:0: warning: "SIGALRM" redefined
 #define SIGALRM SIGFPE

In file included from /usr/include/sys/signal.h:22:0,
                 from /usr/include/signal.h:6,
                 from /usr/include/time.h:178,
                 from /usr/include/sys/time.h:268,
                 from os-autoconf.h:40,
                 from os.h:20,
                 from signals.c:20:
/usr/include/cygwin/signal.h:381:0: note: this is the location of the previous definition
 #define SIGALRM 14 /* alarm clock */

signals.c:25:0: warning: "SIGHUP" redefined
 #define SIGHUP SIGILL

In file included from /usr/include/sys/signal.h:22:0,
                 from /usr/include/signal.h:6,
                 from /usr/include/time.h:178,
                 from /usr/include/sys/time.h:268,
                 from os-autoconf.h:40,
                 from os.h:20,
                 from signals.c:20:
/usr/include/cygwin/signal.h:367:0: note: this is the location of the previous definition
 #define SIGHUP 1 /* hangup */

ar: creazione di secp256k1.a
ar: creazione di aes.a

Make process completed.

Running:

Angelo@DESKTOP-AT0N13V ~/jtest/src
$ ../run/john.exe --format=descrypt-opencl --wordlist=../run/wl.txt ../run/hashes.txt
Device 2: GeForce GTX 1050 Ti
Using default input encoding: UTF-8
Loaded 201083 password hashes with 256 different salts (descrypt-opencl, traditional crypt(3) [DES OpenCL])
Note: This format may be a lot faster with --mask acceleration (see doc/MASK).

Remaining 135666 password hashes with 256 different salts
...
tender           (tender)
49200100         (andrew359)
john0917         (eric1982)
rearaus1         (roberto1)
sarcasti         (sarcasticus)
10299846         (leo5o5)
121170           (donnie)
kbplsgh          (pusher)
42racing         (potterracing)
yoo666           (buull666)
fenster          (reiter)
Shark123         (blkitln)
sherlock         (RWJones)
48862230         (l6kjewfv7se9bg)
pussy4me         (b7jg4fsv59n6utd)
Lilac467         (eric4673)
bujieme          (alinbujie)
faustin          (faustin)
nomelase         (ftzbkejrufg)
niknok           (c2yupupv76hc)
saramydo         (jessesara)
saad450          (gms450)
hongquan         (seu05596)
patatesk         (efecan)
fishgrow         (shipwetdish)
henryshl         (henryshll)
zzzwishm         (zzzwishm)
baggies0         (dogsbody)
40231961         (4023196180a)
tobechi          (hipcat9590)
nonworke         (slacker49)
Alarm clock

test.zip

[magnumripper]

OK so there's more changes needed for properly using OS TIMER on Cygwin. You should probably just revert that. I'll continue my testing on macOS and Linux with OS_TIMER set to 0 and see if I can hit something.

We did have similar problems back in 2013 after I committed stuff that did not properly handle timer emulation but I can't recall any recent change that could cause this.

[magnumripper]

OK, I can reproduce (but not on macOS). I'll fix this soon-ish.

[timajus]

Thanks a lot @magnumripper and thanks for spending time to solve my problem. Just an addition, starting with mask, it starts cracking but after a while it stops showing:

../run/john.exe --mask=?w?d?d?d -min-len=8 -max-len=8 --format=descrypt-opencl --wordlist=../run/wl.txt ../run/hashes.txt
Device 2: GeForce GTX 1050 Ti
Using default input encoding: UTF-8
Loaded 201083 password hashes with 256 different salts (descrypt-opencl, traditional crypt(3) [DES OpenCL])
Remaining 168678 password hashes with 256 different salts
Note: building per-salt kernels. This takes e.g. 2 hours for 4096 salts.
Press 'q' or Ctrl-C to abort, almost any other key for status

...
layao123         (rocklayao)
frank123         (gpfwxfdnegr)
megan123         (bigdaddy-165d05000a168f076547e9f)
werty008         (emuptpdtl6be1g)
mark1221         (horny1-73fde24f1aa7835b9102d33e7)
brian123         (bbpt8fzlrkxcbsw)
tyler112         (tyler505)
boobs999         (savageguy321)
45678912         (sanyub1)
devil832         (Skip6869)
12345678         (karimo123456789)
honda600         (45xq9xibb5jt)
porn9696         (porn6969)
67676767         (williammm)
OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_DES_bs_plug.c:544 - Failed to read buffer buffer_hash_ids.

Alarm clock

Maybe (and hopefully) related to the same issue . All the best and thanks again!

[magnumripper]

Here's a workaround until I nail the exact problem

diff --git a/src/john.c b/src/john.c
index 982f0dd..29465e2 100644
--- a/src/john.c
+++ b/src/john.c
@@ -1565,6 +1565,9 @@ static void john_init(char *name, int argc, char **argv)
        john_register_all(); /* maybe restricted to one format by options */
    common_init();
    sig_init();
+#if !OS_TIMER
+   sig_init_late();
+#endif

    if (!make_check && !(options.flags & (FLG_SHOW_CHK | FLG_STDOUT))) {
        fflush(stdout);
@@ -1788,9 +1791,10 @@ static void john_run(void)
        if (options.flags & FLG_MASK_CHK)
            mask_crk_init(&database);

+#if OS_TIMER
        /* Placed here to disregard load time. */
        sig_init_late();
-
+#endif
        /* Start a resumed session by emitting a status line. */
        if (rec_restored)
            event_pending = event_status = 1;

[magnumripper]

That CL_OUT_OF_RESOURCES thing is another issue (#2653) and unfortunately we don't have anyone working on it. The author got married and never looked back 😆

[timajus]

@magnumripper Thank you very much. I'll make the workaround and see what happens...and about the CL_OUT_OF_RESOURCES...well the author got married so i think for some time he won't get enough spare time ;)

EDIT: maybe something wrong with my environment...anyway... after the john.c patch.. still Alarm clock... I will grab the new version from git...so i get a fresh working copy

$ ../run/john.exe --format=descrypt-opencl --wordlist=../run/wl.txt ../run/hashes.txt
Device 2: GeForce GTX 1050 Ti
Using default input encoding: UTF-8
Loaded 201083 password hashes with 256 different salts (descrypt-opencl, traditional crypt(3) [DES OpenCL])
Remaining 130546 password hashes with 256 different salts

...
o3oavw8g         (neo819usa)
velocett         (bsab33)
lorenzo          (qpdbh9azxr)
trimetal         (lv7nam8q7ak)
pussygal         (wlgtdnrydq6em)
wx19r4           (willie01)
angie1           (lueee42082)
stewart          (jhvzxzdeuntlr)
jollys           (Yogis1)
kierz9           (84vbye6zra)
4w3s0m31         (awesome1)
f1t56g0v         (vpxzgnfakbqrs7)
blessed2         (Vidal1921)
75375395         (soleryiipp)
hawkins          (smspotter)
cbcckp           (Viernes9779)
davelshi         (davelshirt)
elpeo69          (boricuataker69)
nyrfox23         (nyrfox23)
marley07         (marley)
671010           (pauld67-a8a290886bd9f433158922fa)
qui8EreS         (supersweetman)
active61         (cormister10)
b1ll1ona         (picanin482272)
MONEYMAN         (BIGNIGMONEY)
bbrrss           (duckman909)
redfive          (rhino55803)
lexlam           (william0517)
exitluis         (realeo49)
nasoad1a         (nasoad1a)
janefuzz         (janefuzz1)
d69frien         (dfriend)
marcowhi         (9rfncwnyv5qsn)
ma1denca         (ma1dencana)
who_mike         (who_mike)
wsnqer           (nebur15)
yw1psjmk         (kc6766910908)
jaiyen99         (jaiyen999)
Dude83           (gentleman33)
310544Tz         (dimitritz4)
wicked           (Lakebig)
rebirth          (alucard-7cc2f839e29ad17a82427f13)
gaz3131          (garrymills09)
yamaha93         (mmccullen)
alex123          (firehorse)
dosetime         (shhfun)
markod24         (markod24)
chandan          (rallylal)
diabloii         (Nahiek)
oleander         (Cpena51)
aalvarez         (aalvarez)
allenmor         (morrisallen23)
cadets           (sexxxlover69)
kid123           (vsmooth)
Alarm clock

Thanks as always

[timajus]

ok forgot to revert OS TIMER patch... :( now it seems ok. I will make some more tests and i'll let you know (even if i bet you won't me hear so soon LOL)

Thank you very much @magnumripper. Respect!

[solardiz]

the check should be >=, not >, to closer match the previous behavior

FWIW, I realized that neither matches the old behavior. It's 2.5 vs. 3.5 seconds on average, whereas the old behavior was more exactly 3 seconds, and thus was more stable across invocations. Ideally, we'd read a guaranteed-monotonic yet finer-granularity time, such as by using the return value from times() (which JtR already does in a few other places) and comparing the time difference against 3 * clk_tck (where clk_tck comes from bench.h and is supposed to be already initialized when we reach this code).

[magnumripper]

@timajus a final fix is committed now to main tree. Reset all changes and pull latest updates and you should be fine. OTOH the workaround was fine too so you can continue running that until you feel you'd like to update for other reasons.

@solardiz well I don't know quite what to test but the now committed code performs fairly similar to the old one in terms of time ../run/john 320m.in -form:raw-sha1-opencl -w. I'm not sure there would be a point in getting closer to three seconds (which was likely fairly arbitrarily chosen)? I believe we won't have reproducible behaviour anyway as there's some random seed involved.

[solardiz]

OK, let's leave things as-is for now. Thanks!

[timajus]

Just to confirm that "gitting" the main tree and building it works good! Thanks again!

[solardiz]

@timajus Please post a followup to john-users to let others know that the problem is solved in current git. Thanks!

[timajus]

@solardiz: Done and Thanks again to both of you!