GeneAssembly / biosal

biosal is a distributed BIOlogical Sequence Actor Library. THIS IS A MIRROR.
BSD 2-Clause "Simplified" License
6 stars 1 forks source link

Assign local graph stores to visitors to benefit from data locality #845

Closed sebhtml closed 9 years ago

sebhtml commented 9 years ago

Before:

[boisvert@bigmem biosal]$ grep Script log | tail Script: biosal_unitig_visitor Name: 1000371 local sends: 5603 remote sends: 20204 Script: biosal_unitig_visitor Name: 1000439 local sends: 5656 remote sends: 20121 Script: biosal_unitig_visitor Name: 1000475 local sends: 5645 remote sends: 20133 Script: biosal_unitig_visitor Name: 1000463 local sends: 5412 remote sends: 20089 Script: biosal_unitig_visitor Name: 1000391 local sends: 5296 remote sends: 20023 Script: biosal_unitig_visitor Name: 1000479 local sends: 6236 remote sends: 19991 Script: biosal_unitig_visitor Name: 1000467 local sends: 5865 remote sends: 20096 Script: biosal_unitig_visitor Name: 1000443 local sends: 5160 remote sends: 19998 Script: biosal_unitig_visitor Name: 1000487 local sends: 5067 remote sends: 20196 Script: biosal_unitig_visitor Name: 1000455 local sends: 5758 remote sends: 20034

After:

[boisvert@bigmem biosal]$ grep Script log|tail Script: biosal_unitig_visitor Name: 1000455 local sends: 21242 remote sends: 19804 Script: biosal_unitig_visitor Name: 1000435 local sends: 21480 remote sends: 19846 Script: biosal_unitig_visitor Name: 1000467 local sends: 21484 remote sends: 20038 Script: biosal_unitig_visitor Name: 1000475 local sends: 20573 remote sends: 19826 Script: biosal_unitig_visitor Name: 1000463 local sends: 21444 remote sends: 19364 Script: biosal_unitig_visitor Name: 1000439 local sends: 23990 remote sends: 19660 Script: biosal_unitig_visitor Name: 1000479 local sends: 20120 remote sends: 19785 Script: biosal_unitig_visitor Name: 1000483 local sends: 21445 remote sends: 19995 Script: biosal_unitig_visitor Name: 1000487 local sends: 22596 remote sends: 19692 Script: biosal_unitig_visitor Name: 1000471 local sends: 21054 remote sends: 19749

sebhtml commented 9 years ago

test this on Beagle now.

sebhtml commented 9 years ago

tests/Beagle_Cray_XE6/launch-spate-Wisconsin_Restored_Prairie_Soil.sh

Submitted build spate-Wisconsin_Restored_Prairie_Soil-ci-beagle2-36-1152-2015-01-20-17-03-55 (f387db17dd387495d1d8d2ee17c1f42cbeadfaab) 2916067.sdb

in /lustre/beagle2/CompBIO/automated-tests-beagle2

Beagle) grep TIMER spate-Wisconsin_Restored_Prairie_Soil-ci-beagle2-36-1152-2015-01-20-17-03-55.e2916067 TIMER [Load input / Count input data] 1 minutes, 3.843708 seconds TIMER [Load input / Distribute input data] 55.503784 seconds TIMER [Load input] 1 minutes, 59.347496 seconds TIMER [Build assembly graph / Distribute vertices] 8 minutes, 39.777710 seconds TIMER [Build assembly graph / Distribute arcs] 12 minutes, 23.517273 seconds TIMER [Build assembly graph] 21 minutes, 3.294922 seconds TIMER [Visit vertices for unitigs] 17 minutes, 5.664062 seconds TIMER [Walk for unitigs] 66 minutes, 56.456543 seconds TIMER [Total] 107 minutes, 5.098145 seconds