Stability problem with experimental mini-ranks

[sebhtml]

Path: /rap/nne-790-ab/projects/NIH_Human_Microbiome_Project/Samples

Script: scripts/SilverRay-Scalability-SRS011098-mini-ranks/Template.sh-1

Last completed step: $ grep "creating seeds" SRS011098-Ray-MiniRanks-replicate-1.2012-11-12.1.stdout |grep completed|wc -l 112

Versions and flags: $ Ray -version Rank 0: Rank= 0 Size= 1 ProcessIdentifier= 29826 Ray version 2.1.1-devel License for Ray: GNU General Public License version 3 RayPlatform version: 7.0.0-devel License for RayPlatform: GNU Lesser General Public License version 3

MAXKMERLENGTH: 32 Maximum coverage depth stored by CoverageDepth: 4294967295 MAXIMUM_MESSAGE_SIZE_IN_BYTES: 4000 bytes FORCE_PACKING = n ASSERT = y HAVE_LIBZ = y HAVE_LIBBZ2 = y PROFILER_COLLECT = n CLOCK_GETTIME = n linux = y _MSC_VER = n GNUC = y RAY_32_BITS = n RAY_64_BITS = y MPI standard version: MPI 2.1 MPI library: Open-MPI 1.4.5 Compiler: GNU gcc/g++ 4.6.1 With SSE 4.2 With hardware pop count

Last log updates: $ ls -l SRS011098-Ray-MiniRanks-replicate-1.2012-11-12.1.stdout -rw------- 1 sboisver12 nne-790-01 93985898 Nov 13 05:00 SRS011098-Ray-MiniRanks-replicate-1.2012-11-12.1.stdout $ ls -l SRS011098-Ray-MiniRanks-replicate-1.2012-11-12.1.stderr -rw------- 1 sboisver12 nne-790-01 579 Nov 13 15:32 SRS011098-Ray-MiniRanks-replicate-1.2012-11-12.1.stderr

Files written:

$ ls SRS011098-Ray-MiniRanks-replicate-1.2012-11-12.1 CoverageDistributionAnalysis.txt ElapsedTime.txt NumberOfSequences.txt RayPlatform_Version.txt SequencePartition.txt CoverageDistribution.txt FilePartition.txt Plugins RayVersion.txt degreeDistribution.txt NetworkTest.txt RayCommand.txt Scheduling

[sebhtml]

Replicate:

SRS011098-Ray-MiniRanks-replicate-12-.2012-11-09.1.stdout

Same problem.

[sebhtml]

Does the same work in pure MPI mode ?

10103434 sboisver Idle 128 2:00:00:00 Tue Nov 13 15:35:11

[sebhtml]

This works:

mpiexec -n 4 \ -output-filename Test-mini-ranks \ ./Ray \ -mini-ranks-per-rank 7 \ -k 21 -o Test-mini-ranks \ -p SRR001665_1.fastq \ SRR001665_2.fastq \ -p SRR001666_1.fastq \ SRR001666_2.fastq \

Scaffolds >= 500 nt Number: 66 Total length: 4586443 Average: 69491 N50: 134245 Median: 41538 Largest: 265808

[sebhtml]

No problem without mini-ranks:

mpiexec -n 128 Ray \ -o \ SRS011098-Ray-MiniRanks-replicate-1.2012-11-13.1-MPI \ -k \ 31 \ -p \ SRS011098/SRR062051_1.fastq.gz \ SRS011098/SRR062051_2.fastq.gz \ -p \ SRS011098/SRR062052_1.fastq.gz \ SRS011098/SRR062052_2.fastq.gz \ -s \ SRS011098/SRR055676.fastq.gz \ -s \ SRS011098/SRR055756.fastq.gz \ -s \ SRS011098/SRR056894.fastq.gz \ -s \ SRS011098/SRR056977.fastq.gz \ -s \ SRS011098/SRR059326.fastq.gz \ -s \ SRS011098/SRR059327.fastq.gz \ -search \ /rap/nne-790-ab/genomes/EMBL_CDS+GO/EMBL_CDS_Sequences \ -gene-ontology \ /rap/nne-790-ab/genomes/EMBL_CDS+GO/000-Ontologies.txt \ /rap/nne-790-ab/genomes/EMBL_CDS+GO/000-Annotations.txt \ -search \ /rap/nne-790-ab/genomes/RayKmerSearchStuff/last-build/ARDB \ -search \ /rap/nne-790-ab/genomes/RayKmerSearchStuff/last-build/Bacteria-Genomes \ -search \ /rap/nne-790-ab/genomes/RayKmerSearchStuff/last-build/HumanChromosomes \ -search \ /rap/nne-790-ab/genomes/RayKmerSearchStuff/last-build/NCBI-Bacteria_DRAFT \ -search \ /rap/nne-790-ab/genomes/RayKmerSearchStuff/last-build/Viruses-Genomes \ -with-taxonomy \ /rap/nne-790-ab/genomes/taxonomy/last-build/Genome-to-Taxon.tsv \ /rap/nne-790-ab/genomes/taxonomy/last-build/TreeOfLife-Edges.tsv \ /rap/nne-790-ab/genomes/taxonomy/last-build/Taxon-Names.tsv

    NumberOfSequences: 202487723

Network testing: 4 seconds Counting sequences to assemble: 9 minutes, 15 seconds Sequence loading: 34 minutes, 47 seconds K-mer counting: 4 minutes, 39 seconds Coverage distribution analysis: 5 seconds Graph construction: 6 minutes, 29 seconds Null edge purging: 2 minutes, 18 seconds Selection of optimal read markers: 8 minutes, 38 seconds Detection of assembly seeds: 4 minutes, 59 seconds Estimation of outer distances for paired reads: 1 minutes, 14 seconds Bidirectional extension of seeds: 46 minutes, 23 seconds Merging of redundant paths: 16 minutes, 47 seconds Generation of contigs: 10 seconds Scaffolding of contigs: 12 minutes, 22 seconds Counting sequences to search: 7 seconds Graph coloring: 17 minutes, 39 seconds Counting contig biological abundances: 51 seconds Counting sequence biological abundances: 44 minutes, 10 seconds Loading taxons: 3 minutes, 19 seconds Loading tree: 14 seconds Processing gene ontologies: 1 minutes, 6 seconds Computing neighbourhoods: 0 seconds Total: 3 hours, 35 minutes, 36 seconds

Latency:

AverageForAllRanks: 129.922

StandardDeviation: 55.4903

The problem is in the code path of mini-ranks. Possibly something related to messages ;-)

[sebhtml]

I am looking forward to get the running time of this with a latency of 20-40 us instead of the horrible 129 us !

200 M reads assembled and profiled in < 1 hour on 128 cores ? I think the running time with mini-ranks will be 1.5 h

[sebhtml]

Probably fixed in the engine:

https://github.com/sebhtml/RayPlatform/commit/a90ee0dcf87c8d60d8edea859e9f37d6c012b2f0

[sebhtml]

$ msub scripts/SilverRay-Scalability-SRS011098-mini-ranks/Template.sh-1

10110023

[sebhtml]

Works with this patch !