Intermittent benchmark results

[GoogleCodeExporter]

What steps will reproduce the problem?

1. Running command:   pyrit benchmark.

What is the expected output? What do you see instead?

Expect to see benchmark test complete successfully and consistently, as 
per example below. It presently completes on only about 1 of 5 occasions. 

root@Artui:~# pyrit benchmark
Pyrit 0.2.4 (C) 2008, 2009 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3

The ESSID-blobspace seems to be empty; you should create an ESSID...

Running benchmark for about 0 seconds... -

Computed 43531.29 PMKs/s total.
#1: 'CUDA-Device #1 'GeForce GTX 295'': 10609.2 PMKs/s (Occ. 95.0%; RTT 
3.0)
#2: 'CUDA-Device #2 'GeForce GTX 295'': 11198.4 PMKs/s (Occ. 98.9%; RTT 
3.0)
#3: 'CUDA-Device #3 'GeForce GTX 295'': 10342.7 PMKs/s (Occ. 94.9%; RTT 
3.0)
#4: 'CUDA-Device #4 'GeForce GTX 295'': 11046.0 PMKs/s (Occ. 99.0%; RTT 
2.8)
#5: 'CPU-Core (SSE2)': 418.5 PMKs/s (Occ. 96.9%; RTT 3.0)
#6: 'CPU-Core (SSE2)': 420.1 PMKs/s (Occ. 95.8%; RTT 3.0)
#7: 'CPU-Core (SSE2)': 421.8 PMKs/s (Occ. 95.7%; RTT 3.0)
#8: 'CPU-Core (SSE2)': 435.9 PMKs/s (Occ. 98.9%; RTT 3.2)
root@Artui:~#

Failures to complete the benchmark test occur at any one of three stages – 
namely, before Calibrating; whilst Calibrating; or whilst Running with the 
countdown timer. The failure pattern appears to be random and mostly 
results in traceback lines appearing. Very occasionally a benchmark test 
freezes.

Two instances of traceback lines are listed below – the first whilst 
calibrating, the second whilst running.

Pyrit 0.2.4 (C) 2008, 2009 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3

The ESSID-blobspace seems to be empty; you should create an ESSID...

Calibrating... Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 78, in run
    res = self.solve(essid, pwlist)
SystemError: CUDA_ERROR_LAUNCH_FAILED

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 76, in run
    essid, pwlist = self.queue._gather(self.buffersize)
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 368, in 
_gather
    self._check_cores()
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 246, in 
_check_cores
    raise SystemError, "The core '%s' has died unexpectedly." % core
SystemError: The core 'CUDA-Device #4 'GeForce GTX 295'' has died 
unexpectedly.

Traceback (most recent call last):
  File "/usr/bin/pyrit", line 6, in <module>
    pyrit_cli.Pyrit_CLI().initFromArgv()
  File "/usr/lib/python2.5/site-packages/pyrit_cli.py", line 115, in 
initFromArgv
    }.setdefault(commands[0] if len(commands) > 0 else 'help', 
self.print_help)()
  File "/usr/lib/python2.5/site-packages/pyrit_cli.py", line 584, in 
benchmark
    cp.enqueue('foo', pws)
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 303, in 
enqueue
    self._check_cores()
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 246, in 
_check_cores
    raise SystemError, "The core '%s' has died unexpectedly." % core
SystemError: The core 'CUDA-Device #4 'GeForce GTX 295'' has died 
unexpectedly.
root@Artui:~#  

Pyrit 0.2.4 (C) 2008, 2009 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3

The ESSID-blobspace seems to be empty; you should create an ESSID...

Running benchmark for about 52 seconds... / Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 78, in run
    res = self.solve(essid, pwlist)
SystemError: CUDA_ERROR_LAUNCH_FAILED

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 76, in run
    essid, pwlist = self.queue._gather(self.buffersize)
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 368, in 
_gather
    self._check_cores()
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 246, in 
_check_cores
    raise SystemError, "The core '%s' has died unexpectedly." % core
SystemError: The core 'CUDA-Device #4 'GeForce GTX 295'' has died 
unexpectedly.

Traceback (most recent call last):
  File "/usr/bin/pyrit", line 6, in <module>
    pyrit_cli.Pyrit_CLI().initFromArgv()
  File "/usr/lib/python2.5/site-packages/pyrit_cli.py", line 115, in 
initFromArgv
    }.setdefault(commands[0] if len(commands) > 0 else 'help', 
self.print_help)()
  File "/usr/lib/python2.5/site-packages/pyrit_cli.py", line 595, in 
benchmark
    cp.enqueue('foo', pws)
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 303, in 
enqueue
    self._check_cores()
  File "/usr/lib/python2.5/site-packages/cpyrit/cpyrit.py", line 246, in 
_check_cores
    raise SystemError, "The core '%s' has died unexpectedly." % core
SystemError: The core 'CUDA-Device #2 'GeForce GTX 295'' has died 
unexpectedly.
root@Artui:~#   

What version of the product are you using? On what operating system?

Pyrit 0.2.4 and CPyrit 0.2.4 on BT4pr with Nvidia driver 190.32 beta. Also 
cuda-toolkit and cuda-SDK installed from BT repository. Motherboard=Asus 
Rampage II Extreme; CPU=Intel Core i7 920 at 2.8GHz; RAM=6GB; and two 
GeForce GTX 295 cards – being the latest Gigabyte rev 2.0 version with a 
single PCB.

Please provide any additional information below.

Running the pyrit selftest produces a warning “At least some of the 
results seem to be invalid… Do not trust this installation”.

Running Cowpatty (4.6) plus pyrit in passthrough mode against a 996,000 
wordlist has about a 50% success-completion rate.  The last few lines of a 
successful run are shown below.

key no. 930000: untuneable
key no. 940000: vederloos
key no. 950000: verwarntes
key no. 960000: vormvera
key no. 970000: weltordn
key no. 980000: x7aneoscg8
key no. 990000: zigzaguiez

The PSK is "password" [the line has been edited]

996358 passphrases tested in 25.03 seconds:  39811.82 passphrases/second

Nvidia drivers 190.18 beta and 185.18.36 were tried, but the latest beta 
produces a better, albeit still-limited, response with pyrit.

The Grub Bootloader’s menu.lst file has been edited to include uppermem 
524288 and vmalloc=512M. Unlike the vmalloc option, it’s unclear if 
uppermem makes a difference to pyrit’s performance.

Many thanks for looking at this. I look forward to your comments in due 
course.

Original issue reported on code.google.com by fl...@xtra.co.nz on 20 Oct 2009 at 3:21

[GoogleCodeExporter]

I strongly suspect this to be a problem with your hardware. The fact that 
'selftest'
completes without error (e.g. CUDA_ERROR_LAUNCH_FAILED) but reports bad results
points to that.

Try removing one of the GTX295 or one of your RAM-chips from your setup...

Original comment by lukas.l...@gmail.com on 20 Oct 2009 at 6:33

[GoogleCodeExporter]

Many thanks for your response – I’m sure you’re right. 

Both RAM and cards were tested several days ago, to ensure they weren’t 
defective. 

RAM passed a 90 minute memtest and the cards performed to expectations when 
running 
a Windows-based CUDA application (EWSA from Elcomsoft).

Removing a RAM module hasn’t made any difference; and removing a card 
doesn’t seem 
warranted, given their excellent EWSA and (intermittent) pyrit results.

Could the problem lie in the cards’ new architecture ? 

They are both Gigabyte rev 2.0 versions of the GeForce GTX 295 and the 2 
GPU’s on 
each card share a single PCB, whereas the original version had 2 PCB’s, one 
for each 
GPU. Gigabyte designates them as GV-N295-181-B rev 2.0 cards.

Original comment by fl...@xtra.co.nz on 21 Oct 2009 at 9:01

[GoogleCodeExporter]

A new architecture has no such effect.

I still strongly suspect one of your hardware components to be defective. The 
command
'selftest' compares the results that come back from your GPUs to pre-computed 
values
hardwired into Pyrit's sourecode. The error message "At least some results..." 
means
that there were bit-errors in some of the results which passed the whole 
CUDA-driver
stack without being detected. The random crashes (CUDA_ERROR_LAUNCH_FAILED) also
speak for themselves.

You can try the following to try to pinpoint the GPU that causes the problem:

Open /usr/lib/python2.?/site-packages/cpyrit/cpyrit.py and find the line

'for dev_idx, device in enumerate(_cpyrit_cuda.listDevices()):'

modify it like this:

'for dev_idx, device in enumerate(_cpyrit_cuda.listDevices()[0]):'

This will cause Pyrit to only use the first GPU (See list_cores). If you don't
experience any errors (selftest/benchmark/cowpatty), you can increase [0] to 
[1] and
re-test.

Original comment by lukas.l...@gmail.com on 22 Oct 2009 at 10:15

[GoogleCodeExporter]

Your last suggestion was very much appreciated for its clarity and promptness. 

I’ve worked through the exercise twice, just to be sure a “pattern” I had 
noticed, 
was being repeated. My procedure was:

1. Edit the cpyrit.py line by adding [0] as suggested, and then shutdown.
2. Restart and run pyrit’s list_cores, selftest, benchmark, and a passthrough 
test.
   [The selftest, benchmark and passthrough results were all successful]
3. Edit the cyprit.py line by changing [0] to [1] and then shutdown.
4. Restart and run pyrit’s list_cores, selftest, benchmark, and a passthrough 
test.
   [Again, the selftest, benchmark and passthrough results were successful]

At that stage I noticed the same device (#1 GeForce GTX 295) was being seen by 
list_cores, selftest and benchmark.

My original, unedited list_cores output is:

The following cores seem available...
#1:  'CUDA-Device #1 'GeForce GTX 295''
#2:  'CUDA-Device #2 'GeForce GTX 295''
#3:  'CUDA-Device #3 'GeForce GTX 295''
#4:  'CUDA-Device #4 'GeForce GTX 295''
#5:  'CPU-Core (SSE2)'
#6:  'CPU-Core (SSE2)'
#7:  'CPU-Core (SSE2)'
#8:  'CPU-Core (SSE2)'

And an earlier benchmark test listed results for those same cores

The edited [0] list_cores output is:

The following cores seem available...
#1:  'CUDA-Device #1 'GeForce GTX 295''
#2:  'CPU-Core (SSE2)'
#3:  'CPU-Core (SSE2)'
#4:  'CPU-Core (SSE2)'
#5:  'CPU-Core (SSE2)'
#6:  'CPU-Core (SSE2)'
#7:  'CPU-Core (SSE2)'
#8:  'CPU-Core (SSE2)'

Results for selftest and benchmark used the above cores.

The edited [1] list_cores output is:

The following cores seem available...
#1:  'CUDA-Device #1 'GeForce GTX 295''
#2:  'CPU-Core (SSE2)'
#3:  'CPU-Core (SSE2)'
#4:  'CPU-Core (SSE2)'
#5:  'CPU-Core (SSE2)'
#6:  'CPU-Core (SSE2)'
#7:  'CPU-Core (SSE2)'
#8:  'CPU-Core (SSE2)'

Again, the results for selftest and benchmark used the above cores. I 
half-expected 
to see a different CUDA-Device, maybe the #2 GeForce GTX 295 ?

Experimenting further: list_cores results for an edit [2] and then an edit [3] 
also 
picked out the same device, namely the #1 'GeForce GTX 295'.

That prompts me to enquire if the cpyrit.py file can be further edited to pick 
out 
the other GeForce GTX 295 devices #2, #3, and #4 - either in isolation, or 
perhaps 
as a progressive combination ?

Again - many thanks for your help and patience.

Original comment by fl...@xtra.co.nz on 24 Oct 2009 at 1:51

[GoogleCodeExporter]

Blah my fault. It should read

'for dev_idx, device in enumerate(_cpyrit_cuda.listDevices()[:1]):'

Change the [:1] to [:2], [:3] and finally [:4]

Original comment by lukas.l...@gmail.com on 24 Oct 2009 at 6:54

[GoogleCodeExporter]

Thank you for that – the errant “:” made all the difference. 

Apart from the CPU cores:

The [:1] edit revealed:  #1:  'CUDA-Device #1 'GeForce GTX 295''

The [:2] edit revealed:  #1:  'CUDA-Device #1 'GeForce GTX 295''
                         #2:  'CUDA-Device #2 'GeForce GTX 295''

The [:3] edit revealed:  #1:  'CUDA-Device #1 'GeForce GTX 295''
                         #2:  'CUDA-Device #2 'GeForce GTX 295''
                         #3:  'CUDA-Device #3 'GeForce GTX 295''

The [:4] edit revealed:  #1:  'CUDA-Device #1 'GeForce GTX 295''
                         #2:  'CUDA-Device #2 'GeForce GTX 295''
                         #3:  'CUDA-Device #3 'GeForce GTX 295''
                         #4:  'CUDA-Device #4 'GeForce GTX 295''

Edits [:1], [:2] and [:3] passed pyrit’s list_cores, selftest, benchmark, and 
a 
passthrough test.

Edit [:4] completed list_cores OK – completed the selftest with 
“WARNING!”, and 
wouldn’t complete the benchmark or passthrough tests. 

So GPU #4 has been flagged with a defect that I’ll have to get sorted, but in 
the 
interim I can usefully use that edit [:3] option, as the performance of 3 
devices is 
still quite respectable.

Again – let me repeat my appreciation for your considerable help and 
patience. This 
must rate as one of the more-lengthy Issues (now Fixed) occupying space at 
Google 
Code. Is there some practical way I can compensate for this, so that your very 
impressive project continues to flourish ?

Original comment by fl...@xtra.co.nz on 24 Oct 2009 at 11:13

[GoogleCodeExporter]

You should run 'selftest' multiple times to be sure that the defect is isolated 
in
GPU #4...

Closed for now

Original comment by lukas.l...@gmail.com on 25 Oct 2009 at 6:03

• Changed state: Done