OpenCL format failures on GTX 570

[solardiz]

In addition to those in #4343:

Benchmarking: sha256crypt-opencl, crypt(3) $5$ (rounds=5000) [SHA256 OpenCL]... FAILED (cmp_all(1))

Benchmarking: geli-opencl, FreeBSD GELI [PBKDF2-SHA512 AES OpenCL]... FAILED (cmp_all(1))

Benchmarking: krb5pa-md5-opencl, Kerberos 5 AS-REQ Pre-Auth etype 23 [MD4 HMAC-MD5 RC4 OpenCL/mask accel]... Segmentation fault

and that's where my --test --format=opencl run ends for now - probably will see more failures once that one is fixed.

[claudioandre-br]

Benchmarking: sha256crypt-opencl, crypt(3) $5$ (rounds=5000) [SHA256 OpenCL]... FAILED (cmp_all(1))

Is this old bull's board? If so, probably, I have a workaround available in source code (disabled now).

[solardiz]

Right. How can I enable that workaround?

[solardiz]

Benchmarking: krb5pa-md5-opencl, Kerberos 5 AS-REQ Pre-Auth etype 23 [MD4 HMAC-MD5 RC4 OpenCL/mask accel]... Segmentation fault

(gdb) bt
#0  0x00007ffff758d8a8 in clSetKernelArg () from /usr/lib/libOpenCL.so.1
#1  0x00000000007c738e in set_kernel_args () at opencl_krb5pa-md5_fmt_plug.c:155
#2  create_base_clobj () at opencl_krb5pa-md5_fmt_plug.c:222
#3  reset (db=0x41a5920) at opencl_krb5pa-md5_fmt_plug.c:917
#4  0x00000000006e5c86 in benchmark_all () at bench.c:947
#5  0x00000000006fb14a in john_run () at john.c:1795
#6  0x00000000006fd6a1 in main (argc=<optimized out>, argv=<optimized out>) at john.c:2206
(gdb) disass $pc
Dump of assembler code for function clSetKernelArg:
   0x00007ffff758d8a0 <+0>:     test   %rdi,%rdi
   0x00007ffff758d8a3 <+3>:     je     0x7ffff758d8b8 <clSetKernelArg+24>
   0x00007ffff758d8a5 <+5>:     mov    (%rdi),%rax
=> 0x00007ffff758d8a8 <+8>:     mov    0x130(%rax),%rax
   0x00007ffff758d8af <+15>:    jmpq   *%rax
   0x00007ffff758d8b1 <+17>:    nopl   0x0(%rax)
   0x00007ffff758d8b8 <+24>:    mov    $0xffffffd0,%eax
   0x00007ffff758d8bd <+29>:    retq   
End of assembler dump.
(gdb) p/x $rax
$3 = 0x7fff00000000

Line 155 is:

        HANDLE_CLERROR(clSetKernelArg(crypt_kernel, 4, sizeof(buffer_int_keys), (void *) &buffer_int_keys), "Error setting argument 5.");

Commenting it out makes further similar lines succeed, with the format failing because of lacking this argument, as expected.

        //ref_ctr is used as dummy parameter
       buffer_int_keys = clCreateBuffer(context[gpu_id], CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, 4 * mask_int_cand.num_int_cand, mask_int_cand.int_cand ? mask_int_cand.int_cand : (void *)&dummy, &ret_code);
        HANDLE_CLERROR(ret_code, "Error creating buffer argument buffer_int_keys.");

I notice this refers to mask. Indeed, --no-mask makes the format pass test.

I have no idea what the comment about ref_ctr means. There's nothing of that name. We should probably drop that comment.

[solardiz]

(void *)&dummy passes the address of a stack variable, which will be gone. That's probably wrong, but fixing this to e.g. static unsigned int dummy[1000]; doesn't make a difference, which is not surprising given that mask_int_cand.int_cand is non-NULL. Oh, actually with CL_​MEM_​COPY_​HOST_​PTR having dummy on stack was OK.

[solardiz]

The issue appears to be with setting kernel argument 4 to anything at all (tried setting it to buffer_hash_ids like we successfully use for arg 9, but got the same crash at the exact same place). So I tried playing with how it's defined in the kernel, but that didn't help.

[claudioandre-br]

Right. How can I enable that workaround?

I have no idea. I found an old commit where I can't see an NVIDIA bug note that I swear exists.

Well, what you can do is play with:

#define UNROLL_LOOP    0
#define VECTOR_USAGE    1 (or 0)

[1] FTR: https://github.com/openwall/john/commit/82d066177c72a769197acba563b74da7dc55cc6d#diff-eaaf4debd3153877e0fae389a567d7f7

And https://github.com/openwall/john/pull/2221/files#diff-f895213015ee51fa9876c22e2a7003c4R53

[solardiz]

Thanks, Claudio. I easily came up with an adjustment to UNROLL_LOOP (dropped one bit from its value when on sm_2x) and pushed the fix. BTW, I find it really weird and inconvenient that you use decimal for those values. I used hex during my testing, then reverted that value to decimal for consistency - but I think you should convert them all to hex.

[solardiz]

I think we won't bother looking into geli-opencl (unless we also see it fail on current devices/drivers), nor fixing krb5pa-md5-opencl (I saw no obvious bug of ours). I had only opened this issue in case this would help us quickly uncover our bugs that are also relevant on current platforms, but this doesn't appear to be the case. Closing.

[solardiz]

Here's a new one:

Benchmarking: LM-opencl [DES BS OpenCL/mask accel]... 0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1283 - Kernel failed
0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1283 - Kernel failed
FAILED (get_hash[0](1) ca!=5e)

This is reproducible both in a multi-format test and when this format is tested/benchmarked individually. The ca!=5e is reproducible too, although occasionally it's different:

Device 1: GeForce GTX 570
Benchmarking: LM-opencl [DES BS OpenCL/mask accel]... 0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1283 - Kernel failed
FAILED (get_hash[0](0) 58!=ca)

--no-mask doesn't help. LWS=64 GWS=64 doesn't help either.

With this, we have 4 formats listed on this issue, 2 of which we've got to work on this device (sha256crypt-opencl and krb5pa-md5-opencl) and 2 not yet (geli-opencl, LM-opencl).

Edit: Both of the still-failing formats also fail in the same way using a build from 2 months ago, so it's not a very recent regression.

[solardiz]

Dhiru's original geli-opencl works on this device. It broke somewhere with further commits.

[magnumripper]

Dhiru's original geli-opencl works on this device. It broke somewhere with further commits.

That code didn't feature GPU-side post-processing with HMAC-SHA512 and AES. 743238ec made it 3x faster but possibly is what made it no longer work on GTX570.

We have variants of it though!

run/opencl/opencl_aes.h

/*
 * Source the basic AES code. We use a fancy bitsliced version that can
 * do two blocks in parallel except with some devices that are either too
 * buggy to use it, or actually perform slower with it.
 * CPU's seem to generally perform worse with it. Nvidia GPU's love it.
 * macOS may crash just trying to build it.
 */
#if defined(AES_NO_BITSLICE) || cpu(DEVICE_INFO) || (__OS_X__ && gpu_amd(DEVICE_INFO))
#include "opencl_aes_plain.h"
#else
#include "opencl_aes_bitslice.h"
#endif

Try adding something like this to run/opencl/geli_kernel.cl before it sources opencl_aes.h:

#if  SM_MAJOR < something
#define AES_NO_BITSLICE
#endif

[magnumripper]

On a 2080ti using driver 450.66, bench speed drops from 2238K to 1732K using that option. But the important thing is it's completely different AES code, and just might suite card/driver better.

[solardiz]

#if  SM_MAJOR < something

I'll probably try, but we wouldn't know whether the issue was specific to older SM or to older driver version, so we wouldn't know which of these to check to get this to work not only on one old system of ours, but also for users' older systems.

[solardiz]

The LM-opencl issue here is still reproducible on the GTX 570 with our latest code, but somehow the reported get_hash[0](0) or get_hash[0](1) mismatches are now 7 hex digits - that's even weirder than before because these are meant to be indices into the smallest hash table size we support. Also, over multiple runs the same values are sometimes seen on the left hand side and sometimes on the right hand side of the !=:

Device 1: GeForce GTX 570
Benchmarking: LM-opencl [DES BS OpenCL/mask accel]... Loaded 9 hashes with 1 different salts to test db from test vectors
Options used: -I opencl -cl-mad-enable -DSM_MAJOR=2 -DSM_MINOR=0 -cl-nv-verbose -D__GPU__ -DDEVICE_INFO=16402 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=319 -DDEV_VER_MINOR=23 -D_OPENCL_COMPILER -D FULL_UNROLL=1 -D USE_LOCAL_MEM=1 -D WORK_GROUP_SIZE=8 -D OFFSET_TABLE_SIZE=13 -D HASH_TABLE_SIZE=9 -D MASK_ENABLE=0 -D ITER_COUNT=1 -D LOC_0=-1 -D LOC_1=-1 -D LOC_2=-1 -D LOC_3=-1 -D IS_STATIC_GPU_MASK=0 -D CONST_CACHE_SIZE=65536 -D SELECT_CMP_STEPS=2 -D BITMAP_SIZE_BITS_LESS_ONE=262143 -D REQ_BITMAP_BITS=18 $JOHN/opencl/lm_kernel_f.cl
LWS=8 GWS=64 0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1283 - Kernel failed
0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1283 - Kernel failed
FAILED (get_hash[0](1) 3520aca!=36d115e $LM$a9c604d244c4e99d)

Device 1: GeForce GTX 570
Benchmarking: LM-opencl [DES BS OpenCL/mask accel]... Loaded 9 hashes with 1 different salts to test db from test vectors
Options used: -I opencl -cl-mad-enable -DSM_MAJOR=2 -DSM_MINOR=0 -cl-nv-verbose -D__GPU__ -DDEVICE_INFO=16402 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=319 -DDEV_VER_MINOR=23 -D_OPENCL_COMPILER -D FULL_UNROLL=1 -D USE_LOCAL_MEM=1 -D WORK_GROUP_SIZE=8 -D OFFSET_TABLE_SIZE=13 -D HASH_TABLE_SIZE=9 -D MASK_ENABLE=0 -D ITER_COUNT=1 -D LOC_0=-1 -D LOC_1=-1 -D LOC_2=-1 -D LOC_3=-1 -D IS_STATIC_GPU_MASK=0 -D CONST_CACHE_SIZE=65536 -D SELECT_CMP_STEPS=2 -D BITMAP_SIZE_BITS_LESS_ONE=262143 -D REQ_BITMAP_BITS=18 $JOHN/opencl/lm_kernel_f.cl
LWS=8 GWS=64 0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1283 - Kernel failed
FAILED (get_hash[0](0) 1ff0758!=3520aca $LM$cbc501a4d2227783)

Notice how 3520aca changed sides - this suggests the computed hashes might be returned in the wrong order. In other words, the computation is correct and would have matched a test vector, but not the test vector it's expected to match.

[solardiz]

Currently failing on GTX 570 are 7 formats listed in #4343, geli-opencl, and LM-opencl, for a total of 9:

$ grep -B1 FAILED log
Testing: KeePass-opencl [SHA256 AES/Twofish/ChaCha OpenCL]... 0: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_keepass_fmt_plug.c:279 - Copy result back
FAILED (cmp_all(-1))
--
Testing: EncFS-opencl [PBKDF2-SHA1 AES OpenCL]... 0: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_encfs_fmt_plug.c:316 - Copy result back
FAILED (cmp_all(-1))
--
Testing: FVDE-opencl, FileVault 2 [PBKDF2-SHA256 AES OpenCL]... PASS
Testing: geli-opencl, FreeBSD GELI [PBKDF2-SHA512 AES OpenCL]... FAILED (cmp_all(1))
Testing: gpg-opencl, OpenPGP / GnuPG Secret Key [SHA1/SHA2 OpenCL]... (8xOMP) 0: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_gpg_fmt_plug.c:319 - Copy result back
FAILED (cmp_all(-1))
--
Testing: keychain-opencl, Mac OS X Keychain [PBKDF2-SHA1 3DES OpenCL]... 0: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_keychain_fmt_plug.c:252 - Copy result back
FAILED (cmp_all(-1))
--
0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1283 - Kernel failed
FAILED (get_hash[0](1) ca!=5e)
--
Testing: pgpdisk-opencl, PGP Disk / Virtual Disk [SHA1 AES/TwoFish/CAST OpenCL]... 0: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_pgpdisk_fmt_plug.c:290 - Copy result back
FAILED (cmp_all(-1))
Testing: pgpsda-opencl, PGP Self Decrypting Archive [SHA1 CAST OpenCL]... 0: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_pgpsda_fmt_plug.c:244 - Copy result back
FAILED (cmp_all(-1))
--
Testing: ZIP-opencl, WinZip [PBKDF2-SHA1 OpenCL]... 0: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_zip_fmt_plug.c:378 - Copy result back
FAILED (cmp_all(-1))
--
Testing: crypt, generic crypt(3) [?/64]... (8xOMP) PASS
9 out of 509 tests have FAILED

somehow the reported get_hash[0](0) or get_hash[0](1) mismatches are now 7 hex digits

Somehow -v=5 causes this. Without that option, the values are 2 hex digits. It's this weirdness:

                if (options.verbosity > VERB_LEGACY) {
                        // Dump out as much as possible (up to 3 full bytes). This can
                        // help in trying to track down problems, like needing to SWAP
                        // the binary or other issues, when doing BE ports.  Here
                        // PASSWORD_HASH_SIZES is assumed to be 7. This loop will max
                        // out at 6, in that case (i.e. 3 full bytes).
                        int maxi=size;
                        while (maxi+2 < PASSWORD_HASH_SIZES && format->methods.binary_hash[maxi]) {
                                if (format->methods.binary_hash[++maxi] == NULL) {
                                        --maxi;
                                        break;
                                }
                        }
                        if (format->methods.get_hash[maxi] && format->methods.binary_hash[maxi])
                                sprintf(err_buf, "get_hash[%d](%d) %x!=%x %s", size, index,
                                        format->methods.get_hash[maxi](index),
                                        format->methods.binary_hash[maxi](binary),
                                        ciphertext);
                        else
                                sprintf(err_buf, "get_hash[%d](%d) %x!=%x %s", size,
                                        index, format->methods.get_hash[size](index),
                                        format->methods.binary_hash[size](binary),
                                        ciphertext);
                } else
                {
                        sprintf(err_buf, "get_hash[%d](%d) %x!=%x", size,
                                index, format->methods.get_hash[size](index),
                                format->methods.binary_hash[size](binary));
                }

and in particular the inconsistency here:

                                sprintf(err_buf, "get_hash[%d](%d) %x!=%x %s", size, index,
                                        format->methods.get_hash[maxi](index),
                                        format->methods.binary_hash[maxi](binary),
                                        ciphertext);

We pass size into the sprintf, but use maxi as the actual index. Maybe we should have maxi in all of these places, although then we wouldn't know for what original size the failure was. Moreover, this can now end up printing two equal values with != between them if somehow only a smaller-sized get_hash produced a mismatch or if the failure is intermittent. I guess we need to rework this verbose reporting in multiple ways.

[solardiz]

Curiously, trying actual cracking with the LM-opencl format on GTX 570 fails differently (not with "FAILED" per self-test):

Device 1: GeForce GTX 570
Using default input encoding: UTF-8
Using default target encoding: CP850
Loaded 68623 password hashes with no different salts (LM-opencl [DES BS OpenCL])
0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1283 - Kernel failed

Program received signal SIGSEGV, Segmentation fault.
opencl_lm_get_hash_0 (index=<optimized out>) at opencl_lm_b_plug.c:1305
1305            return hash_table_64[hash_ids[3 + 3 * index]] & PH_MASK_0;
(gdb) bt
#0  opencl_lm_get_hash_0 (index=<optimized out>) at opencl_lm_b_plug.c:1305
#1  0x00000000004075ff in is_key_right (dbsalt=0x45316f8, is_test_fmt_case=<optimized out>, plaintext=0x90d9c1 "AAAAAAA", 
    ciphertext=0xd4b5d0 "$LM$cbc501a4d2227783", binary=0x403e4e4, index=0, format=0xd4b400) at formats.c:546
#2  is_key_right (format=0xd4b400, index=0, binary=0x403e4e4, ciphertext=0xd4b5d0 "$LM$cbc501a4d2227783", 
    plaintext=0x90d9c1 "AAAAAAA", is_test_fmt_case=<optimized out>, dbsalt=0x45316f8) at formats.c:506
#3  0x00000000006f77ea in fmt_self_test_body (full_lvl=-1, db=0x403e440, salt_copy=0x403e501, binary_copy=0x403e4e4, 
    format=0xd4b400) at formats.c:1210
#4  fmt_self_test (format=0xd4b400, db=0x403e440) at formats.c:1938
#5  0x00000000006fd938 in john_run () at john.c:1657
#6  0x0000000000700f23 in main (argc=<optimized out>, argv=<optimized out>) at john.c:2029

With --skip-self-test, it gets the same OpenCL failure, but no segfault - it just exits with code 1 after that point. Perhaps there's also an error handling bug there for us to fix if we care.

[solardiz]

Currently failing on GTX 570 are 7 formats listed in #4343, geli-opencl, and LM-opencl, for a total of 9:

FWIW, things have changed a bit since then (on the same old machine and driver):

$ fgrep -B1 FAILED log
Testing: agilekeychain-opencl, 1Password Agile Keychain [PBKDF2-SHA1 AES OpenCL]... PASS
Testing: ansible-opencl, Ansible Vault [PBKDF2-SHA256 HMAC-SHA256 OpenCL]... FAILED (cmp_all(1))
--
Testing: BitLocker-opencl, BitLocker [SHA256 AES OpenCL]... PASS
Testing: bitwarden-opencl, Bitwarden Password Manager [PBKDF2-SHA256 AES OpenCL]... FAILED (cmp_all(1))
--
Testing: FVDE-opencl, FileVault 2 [PBKDF2-SHA256 AES OpenCL]... PASS
Testing: geli-opencl, FreeBSD GELI [PBKDF2-SHA512 AES OpenCL]... FAILED (cmp_all(1))
--
Testing: keychain-opencl, Mac OS X Keychain [PBKDF2-SHA1 3DES OpenCL]... 0: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_keychain_fmt_plug.c:259 - Copy result back
FAILED (cmp_all(-1))
--
0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1305 - Kernel failed
FAILED (get_hash[0](1) ca!=5e)
--
Testing: pfx-opencl, (.pfx, .p12) [PKCS#12 PBE (SHA1/SHA-256/512) OpenCL]... PASS
Testing: pgpdisk-opencl, PGP Disk / Virtual Disk [SHA1 AES/TwoFish/CAST OpenCL]... FAILED (cmp_all(3))
--
Testing: telegram-opencl [PBKDF2-SHA1 AES OpenCL]... PASS
Testing: tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL]... FAILED (cmp_all(1))
--
Testing: ZIP-opencl, WinZip [PBKDF2-SHA1 OpenCL]... PASS
7 out of 91 tests have FAILED

[solardiz]

FWIW, here are the current failures on this ancient system, looks like we got a few more than before. First, one format would interrupt the tests:

$ LWS=64 GWS=64 ./john -te=0 -form=krb5tgs-opencl
Device 1: GeForce GTX 570
Testing: krb5tgs-opencl, Kerberos 5 TGS-REP etype 23 [MD4 HMAC-MD5 RC4 OpenCL]... 0: OpenCL CL_INVALID_VALUE (-30) error in opencl_krb5_tgs_fmt_plug.c:107 - Error creating buffer

Excluding it, the rest complete testing with:

$ LWS=64 GWS=64 ./john -te=0 -form=opencl,-krb5tgs-opencl 2>&1 | tee -a log-opencl

and here are the results:

$ fgrep -B1 FAIL log-opencl
Testing: agilekeychain-opencl, 1Password Agile Keychain [PBKDF2-SHA1 AES OpenCL]... PASS
Testing: ansible-opencl, Ansible Vault [PBKDF2-SHA256 HMAC-SHA256 OpenCL]... FAILED (cmp_all(1))
--
Testing: BitLocker-opencl, BitLocker [SHA256 AES OpenCL]... PASS
Testing: bitwarden-opencl, Bitwarden Password Manager [PBKDF2-SHA256 AES OpenCL]... FAILED (cmp_all(1))
--
Testing: enpass-opencl, Enpass Password Manager [PBKDF2-SHA1/SHA512 AES OpenCL]... PASS
Testing: ethereum-opencl, Ethereum Wallet [PBKDF2-SHA256 Keccak OpenCL]... FAILED (cmp_all(1))
Testing: ethereum-presale-opencl, Ethereum Presale Wallet [PBKDF2-SHA256 AES Keccak OpenCL]... FAILED (cmp_all(1))
Testing: FVDE-opencl, FileVault 2 [PBKDF2-SHA256 AES OpenCL]... FAILED (cmp_all(1))
Testing: geli-opencl, FreeBSD GELI [PBKDF2-SHA512 AES OpenCL]... FAILED (cmp_all(1))
--
Testing: krb5tgs-sha1-opencl, Kerberos 5 TGS-REP etype 17/18 [PBKDF2-SHA1 AES-CTS OpenCL]... PASS
Testing: lp-opencl, LastPass offline [PBKDF2-SHA256 AES OpenCL]... FAILED (cmp_all(1))
Testing: lpcli-opencl, LastPass CLI [PBKDF2-SHA256 AES OpenCL]... FAILED (cmp_all(1))
--
0: OpenCL CL_INVALID_COMMAND_QUEUE (-36) error in opencl_lm_b_plug.c:1307 - Kernel failed
FAILED (get_hash[0](1) ca!=5e)
--
Testing: mysql-sha1-opencl, MySQL 4.1+ [SHA1 OpenCL]... PASS
Testing: notes-opencl, Apple Notes [PBKDF2-SHA256 AES OpenCL]... FAILED (cmp_all(1))
--
Testing: OpenBSD-SoftRAID-opencl [PBKDF2-SHA1 AES OpenCL]... PASS
Testing: PBKDF2-HMAC-SHA256-opencl [PBKDF2-SHA256 OpenCL]... FAILED (cmp_all(1))
--
Testing: pfx-opencl, (.pfx, .p12) [PKCS#12 PBE (SHA1/SHA-256/512) OpenCL]... PASS
Testing: pgpdisk-opencl, PGP Disk / Virtual Disk [SHA1 AES/TwoFish/CAST OpenCL]... FAILED (cmp_all(3))
--
Testing: rar-opencl, RAR3 (length 5) [SHA1 OpenCL AES]... (8xOMP) PASS
Testing: RAR5-opencl [PBKDF2-SHA256 OpenCL]... FAILED (cmp_all(1))
--
Testing: telegram-opencl [PBKDF2-SHA1 AES OpenCL]... PASS
Testing: tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL]... FAILED (cmp_all(1))
--
Testing: ZIP-opencl, WinZip [PBKDF2-SHA1 OpenCL]... PASS
14 out of 95 tests have FAILED

Looks like 1.5 years ago it was 7 of 91, now it's 15 of 96.