search

GeneAssembly / biosal

biosal is a distributed BIOlogical Sequence Actor Library. THIS IS A MIRROR.
BSD 2-Clause "Simplified" License
6 stars 1 forks source link

run strong scaling tests on mira 512x16 1024x16 1536x16 2048x16 #569

Closed sebhtml closed 10 years ago

sebhtml commented 10 years ago

Summary:

512x16 done 1024x16 done 1536x16 to do 2048x16 to do done spate-569-2048x16-2 (there is a memory leak in the arc computation it seems)

Do all 4 of them with the same biosal commit !

sebhtml commented 10 years ago

all in c1 mode and x16 configuration

sebhtml commented 10 years ago

needs #526 first

sebhtml commented 10 years ago

JobName spate-scaling-569-mira-512x16-1

Machine [boisvert@miralac1 biosal]$ hostname miralac1

AllocationStatus [boisvert@miralac1 biosal]$ cbank Name Jobs Charged Available

CompBIO 140 1140203.5 905805.7 JGI-Pilot 0 0.0 0.0

                140       1140203.5        905805.7

Units are in core hours.

Path /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@miralac1 biosal]$ git log | head -n1 commit 2aee87fb6678f41ca7f6b3b1a0bab6ddd7e594ea

Toolchain /soft/compilers/ibmcmp-feb2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@miralac1 biosal-tests]$ cat spate-scaling-569-mira-512x16-1.sh

!/bin/bash

qsub \ -A CompBIO \ -n 512 \ -t 01:00:00 \ -O spate-scaling-569-mira-512x16-1 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-mira-512x16-1

Submission [boisvert@miralac1 biosal-tests]$ ./spate-scaling-569-mira-512x16-1.sh \ Project 'compbio'; job rerouted to queue 'prod-short' 314911

MachineUtilization [boisvert@miralac1 biosal-tests]$ qstat -f -u boisvert JobID JobName User Score WallTime QueuedTime RunTime Nodes State Location Mode Procs Queue StartTime
314911 spate-scaling-569-mira-512x16-1 boisvert 51.0 01:00:00 00:02:05 N/A 512 queued None c1 512 prod-short N/A

ComputationLoad [boisvert@cetuslac1 biosal-tests]$ grep LOAD spate-scaling-569-mira-512x16-1.output|tail [thorium] node/508 COMPUTATION LOAD 0.57 [thorium] node/453 COMPUTATION LOAD 0.52 [thorium] node/274 COMPUTATION LOAD 0.50 [thorium] node/426 COMPUTATION LOAD 0.51 [thorium] node/298 COMPUTATION LOAD 0.50 [thorium] node/384 COMPUTATION LOAD 0.55 [thorium] node/343 COMPUTATION LOAD 0.53 [thorium] node/340 COMPUTATION LOAD 0.49 [thorium] node/479 COMPUTATION LOAD 0.51 [thorium] node/314 COMPUTATION LOAD 0.50

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER spate-scaling-569-mira-512x16-1.output TIMER [Load input / Count input data] 3 minutes, 31.836151 seconds TIMER [Load input / Distribute input data] 1 minutes, 51.007126 seconds TIMER [Load input] 5 minutes, 22.843292 seconds TIMER [Build assembly graph / Distribute vertices] 15 minutes, 31.595215 seconds TIMER [Build assembly graph / Distribute arcs] 25 minutes, 35.647217 seconds TIMER [Build assembly graph] 41 minutes, 7.242432 seconds TIMER [Run actor computation] 46 minutes, 32.458984 seconds

MemoryUtilization [boisvert@cetuslac1 biosal-tests]$ grep "node/20" spate-scaling-569-mira-512x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 9953202176 [boisvert@cetuslac1 biosal-tests]$ grep "node/0" spate-scaling-569-mira-512x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 9889718272 [boisvert@cetuslac1 biosal-tests]$ grep "node/300" spate-scaling-569-mira-512x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 9804378112

Checksum the -o parameter is broken.

GoodComments

Memory usage has improved a lot !

BadComments NeutralComments

sebhtml commented 10 years ago

JobName spate-scaling-569-mira-1024x16-1

Machine [boisvert@miralac1 biosal]$ hostname miralac1

AllocationStatus [boisvert@miralac1 biosal]$ cbank Name Jobs Charged Available

CompBIO 140 1140203.5 905805.7 JGI-Pilot 0 0.0 0.0

                140       1140203.5        905805.7

Units are in core hours.

Path /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@miralac1 biosal]$ git log | head -n1 commit 2aee87fb6678f41ca7f6b3b1a0bab6ddd7e594ea

Toolchain /soft/compilers/ibmcmp-feb2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@miralac1 biosal-tests]$ cat spate-scaling-569-mira-1024x16-1.sh

!/bin/bash

qsub \ -A CompBIO \ -n 1024 \ -t 01:00:00 \ -O spate-scaling-569-mira-1024x16-1 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-mira-1024x16-1

Submission [boisvert@miralac1 biosal-tests]$ ./spate-scaling-569-mira-1024x16-1.sh \ Project 'compbio'; job rerouted to queue 'prod-short' 314912

MachineUtilization [boisvert@miralac1 biosal-tests]$ qstat -f -u boisvert JobID JobName User Score WallTime QueuedTime RunTime Nodes State Location Mode Procs Queue StartTime

314912 spate-scaling-569-mira-1024x16-1 boisvert 51.0 01:00:00 00:00:35 N/A 1024 queued None c1 1024 prod-short N/A
314911 spate-scaling-569-mira-512x16-1 boisvert 51.0 01:00:00 00:02:05 N/A 512 queued None c1 512 prod-short N/A

ComputationLoad [boisvert@cetuslac1 biosal-tests]$ grep LOAD spate-scaling-569-mira-1024x16-1.output|tail [thorium] node/555 COMPUTATION LOAD 0.33 [thorium] node/910 COMPUTATION LOAD 0.41 [thorium] node/523 COMPUTATION LOAD 0.33 [thorium] node/521 COMPUTATION LOAD 0.33 [thorium] node/539 COMPUTATION LOAD 0.33 [thorium] node/538 COMPUTATION LOAD 0.33 [thorium] node/569 COMPUTATION LOAD 0.33 [thorium] node/942 COMPUTATION LOAD 0.41 [thorium] node/926 COMPUTATION LOAD 0.41 [thorium] node/889 COMPUTATION LOAD 0.41

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER spate-scaling-569-mira-1024x16-1.output TIMER [Load input / Count input data] 3 minutes, 2.904434 seconds TIMER [Load input / Distribute input data] 1 minutes, 48.223999 seconds TIMER [Load input] 4 minutes, 51.128448 seconds TIMER [Build assembly graph / Distribute vertices] 11 minutes, 40.000061 seconds TIMER [Build assembly graph / Distribute arcs] 18 minutes, 29.431030 seconds TIMER [Build assembly graph] 30 minutes, 9.431030 seconds TIMER [Run actor computation] 35 minutes, 4.019531 seconds

MemoryUtilization [boisvert@cetuslac1 biosal-tests]$ grep "node/300" spate-scaling-569-mira-1024x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 7008747520 [boisvert@cetuslac1 biosal-tests]$ grep "node/999" spate-scaling-569-mira-1024x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 7221305344 [boisvert@cetuslac1 biosal-tests]$ grep "node/0" spate-scaling-569-mira-1024x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 7546212352 [boisvert@cetuslac1 biosal-tests]$ grep "node/500" spate-scaling-569-mira-1024x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 6996901888 [boisvert@cetuslac1 biosal-tests]$ grep "node/1" spate-scaling-569-mira-1024x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 10380656640 [boisvert@cetuslac1 biosal-tests]$ grep "node/2" spate-scaling-569-mira-1024x16-1.output | grep ByteCount|awk '{print $9}'|sort -n -r | head -n1 7636119552

Checksum -o is broken

GoodComments BadComments

Why is node 1 using 10 GiB anyway ???

10380656640

NeutralComments

sebhtml commented 10 years ago

JobName spate-scaling-569-mira-1536x16-1

Machine mira, the great machine

AllocationStatus [boisvert@miralac1 biosal]$ cbank Name Jobs Charged Available

CompBIO 140 1140203.5 905805.7 JGI-Pilot 0 0.0 0.0

                140       1140203.5        905805.7

Units are in core hours.

Path /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@miralac1 biosal]$ git log | head -n1 commit afcd65b7c2c84a509ae6035fca3ad8efd04936e1

Toolchain /soft/compilers/ibmcmp-feb2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@miralac1 biosal-tests]$ cat spate-scaling-569-mira-1536x16-1.sh

!/bin/bash

qsub \ -A CompBIO \ -n 1536 \ -t 01:00:00 \ -O spate-scaling-569-mira-1536x16-1 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-mira-1536x16-1

Submission [boisvert@miralac1 biosal-tests]$ ./spate-scaling-569-mira-1536x16-1.sh \ Project 'compbio'; job rerouted to queue 'prod-short' 315330

MachineUtilization ComputationLoad RunningTime [boisvert@miralac1 biosal-tests]$ grep TIMER spate-scaling-569-mira-1536x16-1.output TIMER [Load input / Count input data] 3 minutes, 15.213394 seconds TIMER [Load input / Distribute input data] 2 minutes, 7.984810 seconds TIMER [Load input] 5 minutes, 23.198212 seconds TIMER [Build assembly graph / Distribute vertices] 12 minutes, 20.932251 seconds TIMER [Run actor computation] 36 minutes, 6.641846 seconds TIMER [Build assembly graph / Distribute arcs] 18 minutes, 16.664673 seconds TIMER [Build assembly graph] 30 minutes, 37.596924 seconds

MemoryUtilization Checksum GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

JobName spate-scaling-569-mira-2048x16-1

Machine mira, the great machine

AllocationStatus [boisvert@miralac1 biosal]$ cbank Name Jobs Charged Available

CompBIO 140 1140203.5 905805.7 JGI-Pilot 0 0.0 0.0

                140       1140203.5        905805.7

Units are in core hours.

Path /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@miralac1 biosal]$ git log | head -n1 commit afcd65b7c2c84a509ae6035fca3ad8efd04936e1

Toolchain /soft/compilers/ibmcmp-feb2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@miralac1 biosal-tests]$ cat ./spate-scaling-569-mira-2048x16-1.sh

!/bin/bash

qsub \ -A CompBIO \ -n 2048 \ -t 01:00:00 \ -O spate-scaling-569-mira-2048x16-1 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-mira-2048x16-1

Submission [boisvert@miralac1 biosal-tests]$ ./spate-scaling-569-mira-2048x16-1.sh \ Project 'compbio'; job rerouted to queue 'prod-short' 315331

MachineUtilization ComputationLoad RunningTime MemoryUtilization [boisvert@cetuslac1 biosal-tests]$ grep ByteCount spate-scaling-569-mira-2048x16-1.output |awk '{print $9}'|sort -r -n|head 17079205888 17079205888

[boisvert@cetuslac1 biosal-tests]$ cat spate-scaling-569-mira-2048x16-1.output.gz | gunzip | grep ByteCount |grep -v "node/2045"|awk '{print $9}'|sort -r -n|head 5503045632 5503045632

Only node 2045 used 17 GiB, otherwise, the maximum was 5 GiB. [boisvert@miralac1 biosal-tests]$ grep Error spate-scaling-569-mira-2048x16-1.output
DEBUG Error bsal_memory_allocate returned (nil), 63037440 bytes

Checksum GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

after running 512 and 1024

[boisvert@miralac1 ~]$ cbank Name Jobs Charged Available

CompBIO 140 1140203.5 905805.7 JGI-Pilot 0 0.0 0.0

                140       1140203.5        905805.7

Units are in core hours.

sebhtml commented 10 years ago

[boisvert@miralac1 biosal-tests]$ qstat -u boisvert -f JobID JobName User Score WallTime QueuedTime RunTime Nodes State Location Mode Procs Queue StartTime

315331 spate-scaling-569-mira-2048x16-1 boisvert 51.0 01:00:00 00:00:20 N/A 2048 queued None c1 2048 prod-short N/A
315330 spate-scaling-569-mira-1536x16-1 boisvert 51.0 01:00:00 00:01:46 N/A 1536 queued None c1 1536 prod-short N/A

http://status.alcf.anl.gov/mira/activity

sebhtml commented 10 years ago

JobName issue-569-spate-1536x16-99

Machine mira

AllocationStatus [boisvert@cetuslac1 biosal-tests]$ cbank Name Jobs Charged Available

CompBIO 147 216835.4 -216835.4 JGI-Pilot 0 0.0 0.0

                147        216835.4       -216835.4

Units are in core hours.

Path [boisvert@cetuslac1 biosal-tests]$ pwd /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit d0a2661f6d56ae7af58df5af0f8815842683fd76

Toolchain /soft/compilers/ibmcmp-feb2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat issue-569-spate-1536x16-99.sh

!/bin/bash

qsub \ -A CompBIO \ -n 1536 \ -t 01:00:00 \ -O issue-569-spate-1536x16-99 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o issue-569-spate-1536x16-99

Submission [boisvert@miralac1 biosal-tests]$ ./issue-569-spate-1536x16-99.sh \ Project 'compbio'; job rerouted to queue 'prod-short' 316333

MachineUtilization ComputationLoad RunningTime MemoryUtilization Checksum GoodComments BadComments This commit had this bug:

[boisvert@cetuslac1 biosal-tests]$ grep workers issue-569-spate-1536x16-99.output |grep 31592|tail DEBUG manager 142192126 says that spawner 3134575 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134510 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134754 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134904 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134021 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134404 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134143 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134086 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134137 is on a node with 31592 workers DEBUG manager 142192126 says that spawner 3134328 is on a node with 31592 workers

NeutralComments

sebhtml commented 10 years ago

JobName issue-569-spate-2048x16-42

Machine cetus

AllocationStatus [boisvert@cetuslac1 biosal-tests]$ cbank Name Jobs Charged Available

CompBIO 147 216835.4 -216835.4 JGI-Pilot 0 0.0 0.0

                147        216835.4       -216835.4

Units are in core hours.

Path [boisvert@cetuslac1 biosal-tests]$ pwd /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit d0a2661f6d56ae7af58df5af0f8815842683fd76

Toolchain /soft/compilers/ibmcmp-feb2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat issue-569-spate-2048x16-42.sh

!/bin/bash

qsub \ -A CompBIO \ -n 2048 \ -t 01:00:00 \ -O issue-569-spate-2048x16-42 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o issue-569-spate-2048x16-42

Submission not submitted.

MachineUtilization ComputationLoad RunningTime MemoryUtilization Checksum GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

.

sebhtml commented 10 years ago

Our allocation is quite small, so I am not running 1536 and 2048 for now.

sebhtml commented 10 years ago

JobName spate-569-2048x16-1

Machine cetus

AllocationStatus Path [boisvert@cetuslac1 biosal-tests]$ pwd /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit ba418b17d8a9a09400bf09343ccf0948f72bc067

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat spate-569-2048x16-1.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 2048 \ -t 00:30:00 \ -O spate-569-2048x16-1 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-569-2048x16-1

Submission [boisvert@cetuslac1 biosal-tests]$ ./spate-569-2048x16-1.sh 317872

MachineUtilization ComputationLoad RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER spate-569-2048x16-1.output TIMER [Load input / Count input data] 2 minutes, 2.964027 seconds TIMER [Load input / Distribute input data] 2 minutes, 30.560608 seconds TIMER [Load input] 4 minutes, 33.524628 seconds TIMER [Build assembly graph / Distribute vertices] 16 minutes, 52.509705 seconds

MemoryUtilization Ran out of memory. [boisvert@cetuslac1 biosal-tests]$ grep Error spate-569-2048x16-1.output DEBUG Error bsal_memory_allocate returned (nil), 63037440 bytes DEBUG Error bsal_memory_allocate returned (nil), 63037440 bytes DEBUG Error bsal_memory_allocate returned (nil), 63037440 bytes DEBUG Error bsal_memory_allocate returned (nil), 63037440 bytes DEBUG Error bsal_memory_allocate returned (nil), 63037440 bytes

Checksum GoodComments BadComments There is a memory issue in the arc computation code.

Only rank 2045 had such a problem...

[boisvert@cetuslac1 biosal-tests]$ grep ByteCount spate-569-2048x16-1.output|grep -v "node/2045"|awk '{print $9}'|sort -r -n|head -n1 8891981824

[boisvert@cetuslac1 biosal-tests]$ grep ByteCount spate-569-2048x16-1.output|grep "node/2045"|awk '{print $9}'|sort -r -n|head -n1 17090703360

NeutralComments

sebhtml commented 10 years ago

JobName spate-scaling-569-mira-1024x16-3

Machine cetus

AllocationStatus Path /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 801e36aa08d46997cb563bb73dd78887c7b4f47d

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat ./spate-scaling-569-mira-1024x16-3.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 1024 \ -t 01:00:00 \ -O spate-scaling-569-mira-1024x16-3 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-mira-1024x16-3

Submission [boisvert@cetuslac1 biosal-tests]$ ./spate-scaling-569-mira-1024x16-3.sh 318196 http://status.alcf.anl.gov/cetus/activity

MachineUtilization ComputationLoad RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER spate-scaling-569-mira-1024x16-3.output TIMER [Load input / Count input data] 2 minutes, 36.873184 seconds TIMER [Load input / Distribute input data] 1 minutes, 48.446571 seconds TIMER [Load input] 4 minutes, 25.319763 seconds TIMER [Build assembly graph / Distribute vertices] 11 minutes, 12.575195 seconds TIMER [Build assembly graph / Distribute arcs] 19 minutes, 38.218262 seconds TIMER [Build assembly graph] 30 minutes, 50.793457 seconds TIMER [Run actor computation] 35 minutes, 19.814453 seconds

MemoryUtilization Checksum [boisvert@cetuslac1 biosal-tests]$ sha1sum spate-scaling-569-mira-1024x16-3/coverage_distribution.txt-canonical 01a293db48518190038eaddbaed8a47ca0323fc7 spate-scaling-569-mira-1024x16-3/coverage_distribution.txt-canonical

GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

JobName spate-scaling-569-cetus-1024x16-4

Machine cetus

AllocationStatus

Path /projects/CompBIO/Projects/biosal-tests

Commit f4d0616c9c53614dee15d02eae5a27ece9c66603

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat spate-scaling-569-cetus-1024x16-4.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 1024 \ -t 01:00:00 \ -O spate-scaling-569-cetus-1024x16-4 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-cetus-1024x16-4

Submission [boisvert@cetuslac1 biosal-tests]$ ./spate-scaling-569-cetus-1024x16-4.sh 318350

MachineUtilization ComputationLoad [boisvert@cetuslac1 biosal-tests]$ zcat spate-scaling-569-cetus-1024x16-4.output.gz |grep LOAD|grep COMPUTA|tail [thorium] node/603 COMPUTATION LOAD 0.51 [thorium] node/1022 COMPUTATION LOAD 0.55 [thorium] node/841 COMPUTATION LOAD 0.54 [thorium] node/1021 COMPUTATION LOAD 0.55 [thorium] node/523 COMPUTATION LOAD 0.47 [thorium] node/942 COMPUTATION LOAD 0.58 [thorium] node/888 COMPUTATION LOAD 0.58 [thorium] node/1020 COMPUTATION LOAD 0.55 [thorium] node/1006 COMPUTATION LOAD 0.54 [thorium] node/889 COMPUTATION LOAD 0.57

RunningTime [boisvert@cetuslac1 biosal-tests]$ zcat spate-scaling-569-cetus-1024x16-4.output.gz |grep TIMER
TIMER [Load input / Count input data] 1 minutes, 57.415657 seconds TIMER [Load input / Distribute input data] 1 minutes, 55.486313 seconds TIMER [Load input] 3 minutes, 52.901962 seconds TIMER [Build assembly graph / Distribute vertices] 9 minutes, 1.727539 seconds TIMER [Build assembly graph / Distribute arcs] 14 minutes, 22.594604 seconds TIMER [Build assembly graph] 23 minutes, 24.322144 seconds TIMER [Run actor computation] 27 minutes, 20.879028 seconds

MemoryUtilization [boisvert@cetuslac1 biosal-tests]$ zcat spate-scaling-569-cetus-1024x16-4.output.gz |grep ByteCount|awk '{print $9}'|sort -r -n|head 11991523328 11991523328 11911823360 11911823360 11911823360 11911823360 11911823360 11911823360 11911823360 11911823360

Checksum [boisvert@cetuslac1 biosal-tests]$ sha1sum spate-scaling-569-cetus-1024x16-4/coverage_distribution.txt-canonical 01a293db48518190038eaddbaed8a47ca0323fc7 spate-scaling-569-cetus-1024x16-4/coverage_distribution.txt-canonical

GoodComments 27 minutes, 20.879028 seconds < 35 minutes, 19.814453 seconds

BadComments rank 150, 406 (150+256), and 918 (150-256+1024) are all busy

irb(main):004:0> 150-256+1024 => 918 irb(main):005:0> 150+256 => 406 irb(main):006:0> 406-256 => 150 irb(main):007:0> 406+256 => 662 irb(main):008:0> 918-256 => 662 irb(main):009:0> 918+256-1024 => 150

NeutralComments

sebhtml commented 10 years ago

JobName spate-scaling-569-cetus-1024x16-5

Machine cetus

AllocationStatus Path /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit d4dff1fcca1e6f839f7379dca84693f7105e8bde

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat spate-scaling-569-cetus-1024x16-5.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 1024 \ -t 01:00:00 \ -O spate-scaling-569-cetus-1024x16-5 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-cetus-1024x16-5

Submission [boisvert@cetuslac1 biosal-tests]$ ./spate-scaling-569-cetus-1024x16-5.sh 318471

MachineUtilization ComputationLoad none

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER spate-scaling-569-cetus-1024x16-5.output TIMER [Load input / Count input data] 2 minutes, 40.305862 seconds TIMER [Load input / Distribute input data] 1 minutes, 48.780769 seconds TIMER [Load input] 4 minutes, 29.086639 seconds

MemoryUtilization [boisvert@cetuslac1 biosal-tests]$ grep ByteCount spate-scaling-569-cetus-1024x16-5.output|awk '{print $9}'|sort -r -n|head -n1 10926247936

Checksum GoodComments BadComments [boisvert@cetuslac1 biosal-tests]$ grep Error spate-scaling-569-cetus-1024x16-5.output |wc -l 0 [boisvert@cetuslac1 biosal-tests]$ grep Error spate-scaling-569-cetus-1024x16-5.error 2014-08-24 16:13:26.214 (WARN ) [0x4000100bdd0] CET-00040-33771-1024:613495:ibm.runjob.client.Job: most recent RAS event text: DDR Correctable Error Summary : count=79 MCFIR error status: [MEMORY_CE] This bit is set when a memory CE is detected on a non-maintenance memory read op;

NeutralComments

sebhtml commented 10 years ago

JobName spate-scaling-569-cetus-1536x16-7

Machine cetus

AllocationStatus Path /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 7f0902adcb48923c3d902081702bf4f9e78d82c9

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat spate-scaling-569-cetus-1536x16-7.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 1536 \ -t 00:40:00 \ -O spate-scaling-569-cetus-1536x16-7 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-cetus-1536x16-7

Submission [boisvert@cetuslac1 biosal-tests]$ ./spate-scaling-569-cetus-1536x16-7.sh 318489

MachineUtilization ComputationLoad RunningTime Crashed

MemoryUtilization [boisvert@cetuslac1 biosal-tests]$ grep ByteCount spate-scaling-569-cetus-1536x16-7.output|awk '{print $9}'|sort -r -n|head -n1 17086488576 [boisvert@cetuslac1 biosal-tests]$ grep ByteCount spate-scaling-569-cetus-1536x16-7.output|grep -v "node/1533" |awk '{print $9}'|sort -r -n|head -n1 8814923776

Checksum GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

JobName spate-scaling-569-cetus-1024x16-6

Machine cetus

AllocationStatus Path /projects/CompBIO/Projects/biosal-tests

Commit 7f0902adcb48923c3d902081702bf4f9e78d82c9

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat spate-scaling-569-cetus-1024x16-6.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 1024 \ -t 01:00:00 \ -O spate-scaling-569-cetus-1024x16-6 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-scaling-569-cetus-1024x16-6

Submission [boisvert@cetuslac1 biosal-tests]$ ./spate-scaling-569-cetus-1024x16-6.sh 318482

MachineUtilization ComputationLoad [boisvert@cetuslac1 biosal-tests]$ grep COMPUTATION spate-scaling-569-cetus-1024x16-6.output | grep LOAD|tail [thorium] node/1022 COMPUTATION LOAD 0.59 [thorium] node/1006 COMPUTATION LOAD 0.59 [thorium] node/950 COMPUTATION LOAD 0.53 [thorium] node/1012 COMPUTATION LOAD 0.59 [thorium] node/948 COMPUTATION LOAD 0.53 [thorium] node/998 COMPUTATION LOAD 0.57 [thorium] node/1014 COMPUTATION LOAD 0.60 [thorium] node/958 COMPUTATION LOAD 0.53 [thorium] node/1020 COMPUTATION LOAD 0.60 [thorium] node/1004 COMPUTATION LOAD 0.60

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER spate-scaling-569-cetus-1024x16-6.output TIMER [Load input / Count input data] 2 minutes, 29.944946 seconds TIMER [Load input / Distribute input data] 1 minutes, 49.929199 seconds TIMER [Load input] 4 minutes, 19.874146 seconds TIMER [Build assembly graph / Distribute vertices] 9 minutes, 31.964172 seconds TIMER [Build assembly graph / Distribute arcs] 14 minutes, 9.113037 seconds TIMER [Build assembly graph] 23 minutes, 41.077148 seconds TIMER [Run actor computation] 28 minutes, 4.620117 seconds

MemoryUtilization Checksum [boisvert@cetuslac1 biosal-tests]$ sha1sum spate-scaling-569-cetus-1024x16-6/coverage_distribution.txt-canonical 01a293db48518190038eaddbaed8a47ca0323fc7 spate-scaling-569-cetus-1024x16-6/coverage_distribution.txt-canonical

GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

blocked by #610 and #611 and #612

sebhtml commented 10 years ago

JobName spate-569-2048x16-2

Machine cetus

AllocationStatus Path [boisvert@cetuslac1 biosal-tests]$ pwd /projects/CompBIO/Projects/biosal-tests

Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit bd0aa1f9a966c4377f45a374f5d498cf526ff070

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat spate-569-2048x16-2.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 2048 \ -t 00:30:00 \ -O spate-569-2048x16-2 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o spate-569-2048x16-2

Submission [boisvert@cetuslac1 biosal-tests]$ ./spate-569-2048x16-2.sh 318701

MachineUtilization ComputationLoad [boisvert@cetuslac1 biosal-tests]$ grep "COMPUTATION LOAD" spate-569-2048x16-2.output |grep -v " s "|tail [thorium] node/1914 COMPUTATION LOAD 0.48 [thorium] node/1951 COMPUTATION LOAD 0.50 [thorium] node/1772 COMPUTATION LOAD 0.42 [thorium] node/1897 COMPUTATION LOAD 0.49 [thorium] node/1740 COMPUTATION LOAD 0.46 [thorium] node/1741 COMPUTATION LOAD 0.45 [thorium] node/1881 COMPUTATION LOAD 0.50 [thorium] node/1742 COMPUTATION LOAD 0.45 [thorium] node/1948 COMPUTATION LOAD 0.50 [thorium] node/1932 COMPUTATION LOAD 0.50

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER spate-569-2048x16-2.output
TIMER [Load input / Count input data] 2 minutes, 12.618332 seconds TIMER [Load input / Distribute input data] 2 minutes, 29.542252 seconds TIMER [Load input] 4 minutes, 42.160583 seconds TIMER [Build assembly graph / Distribute vertices] 8 minutes, 5.891388 seconds TIMER [Build assembly graph / Distribute arcs] 9 minutes, 41.121033 seconds TIMER [Build assembly graph] 17 minutes, 47.012451 seconds TIMER [Run actor computation] 22 minutes, 35.858154 seconds

MemoryUtilization [boisvert@cetuslac1 biosal-tests]$ grep ByteCount spate-569-2048x16-2.output | awk '{print $9}'|sort -r -n|head -n1 14666194944 [boisvert@cetuslac1 biosal-tests]$ grep ByteCount spate-569-2048x16-2.output | grep -v "node/2045" | awk '{print $9}'|sort -r -n|head -n1 11396468736

Checksum [boisvert@cetuslac1 biosal-tests]$ sha1sum spate-569-2048x16-2/coverage_distribution.txt-canonical 01a293db48518190038eaddbaed8a47ca0323fc7 spate-569-2048x16-2/coverage_distribution.txt-canonical

GoodComments 22 minutes, 35 seconds is the fastest so far.

BadComments

NeutralComments

sebhtml commented 10 years ago

JobName Machine on cetus

AllocationStatus Path Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 5a91ea241e69bd8a02b0911959029fee4c349f9b

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat strong-scaling-569-spate-thorium-cetus-512-99.sh

!/bin/bash

qsub \ -A CompBIO \ -n 512 \ -t 01:00:00 \ -O strong-scaling-569-spate-thorium-cetus-512-99 \ --mode c1 \ spate -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-512-99

Submission [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-512-99.sh 319294

MachineUtilization ComputationLoad RunningTime fail, there was a bug 512x16

1024x16

1536x16

2048x16

MemoryUtilization Checksum GoodComments BadComments there was a bug

NeutralComments

sebhtml commented 10 years ago

JobName Machine AllocationStatus Path Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit d3c6d3c245b94b806cb53f00e9952b03d95a8510

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat strong-scaling-569-spate-thorium-cetus-512x16-101.sh

!/bin/bash

qsub \ -A CompBIO \ -n 512 \ -t 01:00:00 \ -O strong-scaling-569-spate-thorium-cetus-512x16-101 \ --mode c1 \ spate -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-512x16-101

Submission [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-512x16-101.sh 319354 [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-1024x16-101.sh 319368 [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-1536x16-101.sh 319403 [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-2048x16-101.sh 319497

MachineUtilization ComputationLoad RunningTime 512x16 [boisvert@cetuslac1 biosal-tests]$ grep TIMER strong-scaling-569-spate-thorium-cetus-512x16-101.output TIMER [Load input / Count input data] 3 minutes, 6.749069 seconds TIMER [Load input / Distribute input data] 1 minutes, 53.909935 seconds TIMER [Load input] 5 minutes, 0.658997 seconds TIMER [Build assembly graph / Distribute vertices] 17 minutes, 19.924805 seconds TIMER [Build assembly graph / Distribute arcs] 24 minutes, 9.365479 seconds TIMER [Build assembly graph] 41 minutes, 29.290283 seconds TIMER [Run actor computation] 46 minutes, 32.613525 seconds

1024x16 [boisvert@cetuslac1 biosal-tests]$ grep TIMER strong-scaling-569-spate-thorium-cetus-1024x16-101.output TIMER [Load input / Count input data] 2 minutes, 31.875290 seconds TIMER [Load input / Distribute input data] 1 minutes, 39.430809 seconds TIMER [Load input] 4 minutes, 11.306091 seconds TIMER [Build assembly graph / Distribute vertices] 12 minutes, 11.156921 seconds TIMER [Build assembly graph / Distribute arcs] 17 minutes, 10.203003 seconds TIMER [Build assembly graph] 29 minutes, 21.359985 seconds TIMER [Run actor computation] 33 minutes, 37.020142 seconds

1536x16 [boisvert@cetuslac1 biosal-tests]$ grep TIMER strong-scaling-569-spate-thorium-cetus-1536x16-101.output TIMER [Load input / Count input data] 2 minutes, 8.456497 seconds TIMER [Load input / Distribute input data] 2 minutes, 5.977280 seconds TIMER [Load input] 4 minutes, 14.433777 seconds TIMER [Build assembly graph / Distribute vertices] 11 minutes, 47.464294 seconds TIMER [Build assembly graph / Distribute arcs] 15 minutes, 58.214905 seconds TIMER [Build assembly graph] 27 minutes, 45.679199 seconds TIMER [Run actor computation] 32 minutes, 7.436401 seconds

2048x16

MemoryUtilization Checksum [boisvert@cetuslac1 biosal-tests]$ sha1sum ./strong-scaling-569-spate-thorium-cetus-512x16-101/coverage_distribution.txt-canonical 01a293db48518190038eaddbaed8a47ca0323fc7 ./strong-scaling-569-spate-thorium-cetus-512x16-101/coverage_distribution.txt-canonical

[boisvert@cetuslac1 biosal-tests]$ sha1sum ./strong-scaling-569-spate-thorium-cetus-1024x16-101/coverage_distribution.txt-canonical 01a293db48518190038eaddbaed8a47ca0323fc7 ./strong-scaling-569-spate-thorium-cetus-1024x16-101/coverage_distribution.txt-canonical

GoodComments BadComments I forgot to use --env PAMID_THREAD_MULTIPLE=1 \

NeutralComments

sebhtml commented 10 years ago

With "--env PAMID_THREAD_MULTIPLE=1 \"

JobName Machine AllocationStatus Path Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 4487417cb20689ed119a5a78998373a51bc1f2bf

Toolchain /soft/compilers/ibmcmp-may2014/vac/bg/12.1/bin/.orig/bgxlc_r:

Script [boisvert@cetuslac1 biosal-tests]$ cat strong-scaling-569-spate-thorium-cetus-2048x16-200.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 2048 \ -t 00:30:00 \ -O strong-scaling-569-spate-thorium-cetus-2048x16-200 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-2048x16-200

Submission [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-2048x16-200.sh 319510

MachineUtilization ComputationLoad RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER strong-scaling-569-spate-thorium-cetus-2048x16-200.output TIMER [Load input / Count input data] 2 minutes, 7.121826 seconds TIMER [Load input / Distribute input data] 2 minutes, 40.635452 seconds TIMER [Load input] 4 minutes, 47.757294 seconds TIMER [Build assembly graph / Distribute vertices] 11 minutes, 27.023804 seconds

MemoryUtilization Checksum GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

JobName Machine AllocationStatus Path Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 674f6a98850f1cb9f254ebd1fbfcf872fa9a913f

Toolchain Script [boisvert@cetuslac1 biosal-tests]$ cat strong-scaling-569-spate-thorium-cetus-2048x16-202.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 2048 \ -t 00:30:00 \ -O strong-scaling-569-spate-thorium-cetus-2048x16-202 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-2048x16-202

Submission [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-2048x16-202.sh 319668

MachineUtilization ComputationLoad RunningTime MemoryUtilization Checksum GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

2048x16 node 14 load

random numbers on BGQ are not very good (or I am not using the API correctly.)

2 actors get the same name... window/290205210 generated 3194736 kmers from 82200 entries (5 blocks), sent 5 messages to consumer bsal_assembly_arc_kernel/290205210 is now active

sebhtml commented 10 years ago

JobName Machine AllocationStatus Path Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 4d4b4224e844faef4485a01ac9999872caad6f44

Toolchain Script [boisvert@cetuslac1 biosal-tests]$ cat strong-scaling-569-spate-thorium-cetus-2048x16-208.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 2048 \ -t 00:30:00 \ -O strong-scaling-569-spate-thorium-cetus-2048x16-208 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-2048x16-208

Submission [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-2048x16-208.sh 320026

MachineUtilization ComputationLoad [boisvert@cetuslac1 biosal-tests]$ grep "COMPUTATION LOAD" strong-scaling-569-spate-thorium-cetus-2048x16-208.output |grep -v " s " | tail [thorium] node/1658 COMPUTATION LOAD 0.47 [thorium] node/1959 COMPUTATION LOAD 0.48 [thorium] node/1711 COMPUTATION LOAD 0.46 [thorium] node/1536 COMPUTATION LOAD 0.47 [thorium] node/1641 COMPUTATION LOAD 0.47 [thorium] node/1904 COMPUTATION LOAD 0.48 [thorium] node/1983 COMPUTATION LOAD 0.47 [thorium] node/1866 COMPUTATION LOAD 0.47 [thorium] node/1994 COMPUTATION LOAD 0.47 [thorium] node/1948 COMPUTATION LOAD 0.46 node 35 load

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER strong-scaling-569-spate-thorium-cetus-2048x16-208.output TIMER [Load input / Count input data] 2 minutes, 8.520172 seconds TIMER [Load input / Distribute input data] 2 minutes, 54.786865 seconds TIMER [Load input] 5 minutes, 3.307037 seconds TIMER [Build assembly graph / Distribute vertices] 9 minutes, 12.098389 seconds TIMER [Build assembly graph / Distribute arcs] 10 minutes, 35.075439 seconds TIMER [Build assembly graph] 19 minutes, 47.173828 seconds TIMER [Run actor computation] 24 minutes, 57.615967 seconds

MemoryUtilization [boisvert@cetuslac1 biosal-tests]$ grep ByteCount strong-scaling-569-spate-thorium-cetus-2048x16-208.output|awk '{print $9}'|sort -r -n|head 12397211648 12160864256 12083757056 12081168384 11964559360 11944034304 11936497664 11924365312 11921555456 11888467968

Checksum [boisvert@cetuslac1 biosal-tests]$ sha1sum strong-scaling-569-spate-thorium-cetus-2048x16-208/coverage_distribution.txt-canonical 01a293db48518190038eaddbaed8a47ca0323fc7 strong-scaling-569-spate-thorium-cetus-2048x16-208/coverage_distribution.txt-canonical

GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

JobName Machine AllocationStatus Path Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 4d4b4224e844faef4485a01ac9999872caad6f44

Toolchain Script [boisvert@cetuslac1 biosal-tests]$ cat ./strong-scaling-569-spate-thorium-cetus-1536x16-208.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 1536 \ -t 00:40:00 \ -O strong-scaling-569-spate-thorium-cetus-1536x16-208 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-1536x16-208

Submission [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-1536x16-208.sh 320154

MachineUtilization ComputationLoad node 11 load

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER strong-scaling-569-spate-thorium-cetus-1536x16-208.output TIMER [Load input / Count input data] 2 minutes, 35.428085 seconds TIMER [Load input / Distribute input data] 2 minutes, 17.787552 seconds TIMER [Load input] 4 minutes, 53.215637 seconds TIMER [Build assembly graph / Distribute vertices] 8 minutes, 14.590118 seconds TIMER [Build assembly graph / Distribute arcs] 11 minutes, 7.709412 seconds TIMER [Build assembly graph] 19 minutes, 22.299561 seconds TIMER [Run actor computation] 24 minutes, 21.472534 seconds

MemoryUtilization Checksum GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

JobName Machine AllocationStatus Path Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 4d4b4224e844faef4485a01ac9999872caad6f44

Toolchain Script [boisvert@cetuslac1 biosal-tests]$ cat strong-scaling-569-spate-thorium-cetus-1024x16-208.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 1024 \ -t 00:40:00 \ -O strong-scaling-569-spate-thorium-cetus-1024x16-208 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-1024x16-208

Submission [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-1024x16-208.sh 320220

MachineUtilization ComputationLoad node 11 load

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER strong-scaling-569-spate-thorium-cetus-1024x16-208.output TIMER [Load input / Count input data] 2 minutes, 37.128494 seconds TIMER [Load input / Distribute input data] 2 minutes, 0.012146 seconds TIMER [Load input] 4 minutes, 37.140625 seconds TIMER [Build assembly graph / Distribute vertices] 8 minutes, 58.439941 seconds TIMER [Build assembly graph / Distribute arcs] 13 minutes, 15.350098 seconds TIMER [Build assembly graph] 22 minutes, 13.790039 seconds TIMER [Run actor computation] 26 minutes, 54.968994 seconds

MemoryUtilization Checksum GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

JobName Machine AllocationStatus Path Commit [boisvert@cetuslac1 biosal]$ git log | head -n1 commit 4d4b4224e844faef4485a01ac9999872caad6f44

Toolchain Script [boisvert@cetuslac1 biosal-tests]$ cat ./strong-scaling-569-spate-thorium-cetus-512x16-208.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 512 \ -t 00:40:00 \ -O strong-scaling-569-spate-thorium-cetus-512x16-208 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-512x16-208 [boisvert@cetuslac1 biosal-tests]$ cat ./strong-scaling-569-spate-thorium-cetus-512x16-209.sh

!/bin/bash

qsub \ --env PAMID_THREAD_MULTIPLE=1 \ -A CompBIO \ -n 512 \ -t 01:00:00 \ -O strong-scaling-569-spate-thorium-cetus-512x16-209 \ --mode c1 \ spate -print-load -threads-per-node 16 -k 43 Iowa_Continuous_Corn/*.fastq -o strong-scaling-569-spate-thorium-cetus-512x16-209

Submission [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-512x16-208.sh 320323 [boisvert@cetuslac1 biosal-tests]$ ./strong-scaling-569-spate-thorium-cetus-512x16-209.sh 320371

MachineUtilization ComputationLoad [boisvert@cetuslac1 biosal-tests]$ grep LOAD strong-scaling-569-spate-thorium-cetus-512x16-209.output |grep COMP|grep -v " s "|tail [thorium] node/510 COMPUTATION LOAD 0.74 [thorium] node/401 COMPUTATION LOAD 0.69 [thorium] node/373 COMPUTATION LOAD 0.68 [thorium] node/313 COMPUTATION LOAD 0.68 [thorium] node/478 COMPUTATION LOAD 0.72 [thorium] node/403 COMPUTATION LOAD 0.69 [thorium] node/366 COMPUTATION LOAD 0.68 [thorium] node/306 COMPUTATION LOAD 0.68 [thorium] node/434 COMPUTATION LOAD 0.69 [thorium] node/486 COMPUTATION LOAD 0.73 node 0 load

RunningTime [boisvert@cetuslac1 biosal-tests]$ grep TIMER strong-scaling-569-spate-thorium-cetus-512x16-209.output TIMER [Load input / Count input data] 3 minutes, 6.015656 seconds TIMER [Load input / Distribute input data] 2 minutes, 5.853203 seconds TIMER [Load input] 5 minutes, 11.868866 seconds TIMER [Build assembly graph / Distribute vertices] 13 minutes, 16.326904 seconds TIMER [Build assembly graph / Distribute arcs] 21 minutes, 40.002686 seconds TIMER [Build assembly graph] 34 minutes, 56.329590 seconds TIMER [Run actor computation] 40 minutes, 10.927979 seconds

MemoryUtilization Checksum [boisvert@cetuslac1 biosal-tests]$ sha1sum strong-scaling-569-spate-thorium-cetus-512x16-209/coverage_distribution.txt-canonical 01a293db48518190038eaddbaed8a47ca0323fc7 strong-scaling-569-spate-thorium-cetus-512x16-209/coverage_distribution.txt-canonical

GoodComments BadComments NeutralComments

sebhtml commented 10 years ago

Check public/625/ in http://biosal.s3.amazonaws.com/index.html