search

GeneAssembly / biosal

biosal is a distributed BIOlogical Sequence Actor Library. THIS IS A MIRROR.
BSD 2-Clause "Simplified" License
6 stars 1 forks source link

try Spate on Edison (Iowa Continuous Corn) #822

Closed sebhtml closed 9 years ago

sebhtml commented 9 years ago

boisvert@edison01:~> getnim -Uboisvert m1523 505912.81 ACTV

copy started to /project/projectdirs/m1523/Data/Iowa_Continuous_Corn

sebhtml commented 9 years ago

JobName Goal First job on Edison

Machine Edison at NERSC

AllocationStatus boisvert@edison12:/global/u2/b/boisvert> getnim -Uboisvert m1523 768017.13 ACTV

Path /project/projectdirs/m1523/Jobs

Commit 7fc1223b4b1cd79006c5c6bf8ca82c38ff8ad883 make CC=cc applications/spate_metagenome_assembler/spate

Toolchain Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.2.144 Build 20140120

Script boisvert@edison12:/project/projectdirs/m1523/Jobs> cat spate-iowa-continuous-corn-soil-1.pbs

!/bin/bash

PBS -N spate-iowa-continuous-corn-soil-1

PBS -A m1523

PBS -l walltime=2:00:00

PBS -l mppwidth=6144

PBS -q regular

cd $PBS_O_WORKDIR

export MPICH_NEMESIS_ASYNC_PROGRESS=1 export MPICH_MAX_THREAD_SAFETY=multiple

aprun -n 256 -N 1 -d 23 -r 1 \ spate-iowa-continuous-corn-soil-1.spate -threads-per-node 23 -print-load \ -k 33 Iowa_Continuous_Corn/*.fastq -o spate-iowa-continuous-corn-soil-1 \ -freopen-stdout > spate-iowa-continuous-corn-soil-1.stdout

Submission boisvert@edison12:/project/projectdirs/m1523/Jobs> qsub spate-iowa-continuous-corn-soil-1.pbs 2083375.edique02

queue charge factor: 1.0 machine charge factor: 2.0 time: 2 hours node count: 256 cores: 256 * 24 irb(main):001:0> 2 * 256 * 24 * 1.0 * 2.0 => 24576.0

24576 MPP hours MachineUtilization ComputationLoad RunningTime boisvert@edison12:/project/projectdirs/m1523/Jobs> cat spate-iowa-continuous-corn-soil-1.e2083375 aprun: file spate-iowa-continuous-corn-soil-1.spate not found aprun: Exiting due to errors. Application aborted boisvert@edison12:/project/projectdirs/m1523/Jobs> ls spate-iowa-continuous-corn-soil-1.spate spate-iowa-continuous-corn-soil-1.spate

MemoryUtilization Checksum GoodComments BadComments NeutralComments

sebhtml commented 9 years ago

JobName Goal Figure out how to pick up the executable. On Beagle, it workerd without './'.

Machine AllocationStatus Path Commit Toolchain Script boisvert@edison12:/project/projectdirs/m1523/Jobs> cat spate-iowa-continuous-corn-soil-2.pbs

!/bin/bash

PBS -N spate-iowa-continuous-corn-soil-2

PBS -A m1523

PBS -l walltime=2:00:00

PBS -l mppwidth=6144

PBS -q regular

cd $PBS_O_WORKDIR

export MPICH_NEMESIS_ASYNC_PROGRESS=1 export MPICH_MAX_THREAD_SAFETY=multiple

aprun -n 256 -N 1 -d 23 -r 1 \ ./spate-iowa-continuous-corn-soil-2.spate -threads-per-node 23 -print-load \ -k 33 Iowa_Continuous_Corn/*.fastq -o spate-iowa-continuous-corn-soil-2 \ -freopen-stdout > spate-iowa-continuous-corn-soil-2.stdout

boisvert@edison12:/project/projectdirs/m1523/Jobs> ls -lh ./spate-iowa-continuous-corn-soil-2.spate -rwxr-x--- 1 boisvert m1523 8,8M 23 nov 17:33 ./spate-iowa-continuous-corn-soil-2.spate

Submission boisvert@edison12:/project/projectdirs/m1523/Jobs> qsub spate-iowa-continuous-corn-soil-2.pbs 2093511.edique02

MachineUtilization ComputationLoad RunningTime boisvert@edison12:/project/projectdirs/m1523/Jobs> cat spate-iowa-continuous-corn-soil-2.00*.txt|grep TIMER TIMER [Load input / Count input data] 21.429867 seconds TIMER [Load input / Distribute input data] 25.910631 seconds TIMER [Load input] 47.340496 seconds

TIMER [Build assembly graph / Distribute vertices] 2 minutes, 0.706993 secondscore_manager/1021181 dies TIMER [Build assembly graph / Distribute arcs] 4 minutes, 30.089874 seconds TIMER [Build assembly graph] 6 minutes, 30.796875 seconds

TIMER [Visit vertices for unitigs] 54 minutes, 0.475098 seconds TIMER [Walk for unitigs] 20 minutes, 53.441650 seconds

TIMER [Total] 82 minutes, 15.549316 seconds

MemoryUtilization thorium_node: node/192 METRICS AliveActorCount: 221 ByteCount: 18323357696 / 67657900032

Checksum GoodComments http://lists.cels.anl.gov/pipermail/biosal/2014-November/000108.html

boisvert@edison12:/project/projectdirs/m1523/Jobs> grep '>' spate-iowa-continuous-corn-soil-2/unitigs.fasta |awk '{print $2}'|sed 's/length=//g'|sort -r -n|head 20117 20117 20117 20117 20117 20117 20117 20117 11302 11302

boisvert@edison12:/project/projectdirs/m1523/Jobs> ls -l spate-iowa-continuous-corn-soil-2/unitigs.fasta -rw-r----- 1 boisvert m1523 2751274529 24 nov 10:14 spate-iowa-continuous-corn-soil-2/unitigs.fasta

boisvert@edison12:/project/projectdirs/m1523/Jobs> grep GRAPH spate-iowa-continuous-corn-soil-2/_txt
spate-iowa-continuous-corn-soil-2/spate-iowa-continuous-corn-soil-2.00253.txt:GRAPH -> 148375705714 vertices, 298256036296 vertex observations, and 146235667225 arcs. boisvert@edison12:/project/projectdirs/m1523/Jobs> grep flag spate-iowa-continuous-corn-soil-2/_txt
spate-iowa-continuous-corn-soil-2/spate-iowa-continuous-corn-soil-2.00252.txt:DEBUG biosal_unitig_manager/1038076 processed_vertices 17522841967 vertices_with_unitig_flag 15211334979

irb(main):004:0> (148375705714 - 113330021780)/2 => 17522841967

BadComments NeutralComments

sebhtml commented 9 years ago

.: paste Edison link here from the list

sebhtml commented 9 years ago

JobName Goal

Machine Edison

AllocationStatus boisvert@edison12:/global/u2/b/boisvert/m1523/Jobs/biosal> getnim -U$(whoami) m1523 767979.58 ACTV

Path /project/projectdirs/m1523/Jobs

Commit feb224e93a

The job takes forever to start. I updated the build: e3a40915709c New build artifact: boisvert@edison12:/global/u2/b/boisvert/m1523/Jobs/biosal> cp applications/spate_metagenome_assembler/spate /project/projectdirs/m1523/Jobs/spate-edison-nersc-512x24-2014-11-24-1.spate boisvert@edison12:/global/u2/b/boisvert/m1523/Jobs/biosal> sync

-> 86ca3ed049e

no patch

Toolchain Intel

Script boisvert@edison12:/project/projectdirs/m1523/Jobs> cat spate-edison-nersc-512x24-2014-11-24-1.pbs

!/bin/bash

PBS -N spate-edison-nersc-512x24-2014-11-24-1

PBS -A m1523

PBS -l walltime=4:00:00

PBS -l mppwidth=12288

PBS -q regular

cd $PBS_O_WORKDIR

export MPICH_NEMESIS_ASYNC_PROGRESS=1 export MPICH_MAX_THREAD_SAFETY=multiple

aprun -n 512 -N 1 -d 23 -r 1 \ ./spate-edison-nersc-512x24-2014-11-24-1.spate -threads-per-node 23 -print-load \ -k 33 Iowa_Continuous_Corn/*.fastq -o spate-edison-nersc-512x24-2014-11-24-1 \

spate-edison-nersc-512x24-2014-11-24-1.stdout

Submission boisvert@edison12:/project/projectdirs/m1523/Jobs> qsub spate-edison-nersc-512x24-2014-11-24-1.pbs 2098464.edique02

2014-11-25... still queued 2014-11-26... queued, rank 167 2014-11-27... Q, rank 67 2014-11-28 ... Q, rank 0041

MachineUtilization ComputationLoad RunningTime boisvert@edison12:/project/projectdirs/m1523/Jobs> grep TIMER spate-edison-nersc-512x24-2014-11-24-1.stdout TIMER [Load input / Count input data] 22.724026 seconds TIMER [Load input / Distribute input data] 23.425491 seconds TIMER [Load input] 46.149517 seconds TIMER [Build assembly graph / Distribute vertices] 1 minutes, 10.660408 seconds TIMER [Build assembly graph / Distribute arcs] 2 minutes, 32.636459 seconds TIMER [Build assembly graph] 3 minutes, 43.296860 seconds

The charge should be at most:

irb(main):002:0> 4_512_24_1.0_2.0 => 98304.0

MemoryUtilization

thorium_node: node/189 METRICS AliveActorCount: 5149 ByteCount: 13841072128 / 67657900032

Checksum GoodComments boisvert@edison12:/project/projectdirs/m1523/Jobs> grep Total spate-edison-nersc-512x24-2014-11-24-1.stdout
DEBUG controller 1003519: Partition Total: 2228341042, block_size: 65536, blocks: 34002

boisvert@edison12:/project/projectdirs/m1523/Jobs> grep GRAPH spate-edison-nersc-512x24-2014-11-24-1.stdout GRAPH -> 148375705714 vertices, 298256036296 vertex observations, and 140978080824 arcs.

BadComments NeutralComments

sebhtml commented 9 years ago

Let's focus on Iowa Native Prairier #830

When the code is ready, we'll assemble them all anyway.