run on 512 x 16 on mira on Iowa Continuous Corn

[sebhtml]

[boisvert@miralac1 biosal-tests]$ ./test-mira-512x16-1.sh \ Project 'compbio'; job rerouted to queue 'prod-short' 302604

[sebhtml]

/gpfs/mira-fs1/projects/CompBIO/Projects/biosal-tests

[sebhtml]

on cetus: #440 512x16

[sebhtml]

302604 test-mira-512x16-1 boisvert 52.6 01:00:00 06:16:52 N/A 512 queued None c1 512 prod-short N/A

[sebhtml]

version:

[boisvert@miralac1 biosal]$ git log|head -n1 commit a62f6652ee24bed97e2d55e79234099bcc57b5e8

Setup:

Workers: irb(main):001:0> 512*15 => 7680

Node setup (Thorium):

It uses MPI (not PAMI since it is not ready): TRANSPORT rank: 404 size: 512 implementation: MPI

Thorium message:

booted node 441 (512 nodes), threads: 16, workers: 15, pacing: 1 Efficiency: ``` [boisvert@miralac1 biosal-tests]$ grep efficiency test-mira-512x16-1.output|tail node 476 efficiency: 0.32 node 418 efficiency: 0.32 node 390 efficiency: 0.32 node 288 efficiency: 0.32 node 477 efficiency: 0.32 node 417 efficiency: 0.32 node 441 efficiency: 0.32 node 296 efficiency: 0.32 node 426 efficiency: 0.32 node 324 efficiency: 0.32 ``` Timers: ``` [boisvert@miralac1 biosal-tests]$ grep TIMER test-mira-512x16-1.output TIMER 11 minutes, 30.284302 seconds TIMER 7 minutes, 44.152588 seconds TIMER 19 minutes, 14.436890 seconds TIMER 16 minutes, 3.674500 seconds TIMER 35 minutes, 26.579834 seconds ``` Load during streaming: Assessment: It looks pretty damn good actually ``` LOAD EPOCH 1215 s node/73 14.95/15 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 LOAD EPOCH 1215 s node/204 14.48/15 (0.97) 0.94 0.94 1.00 0.94 1.00 1.00 0.94 1.00 0.94 0.94 0.99 1.00 0.94 0.94 0.99 LOAD EPOCH 1215 s node/346 11.43/15 (0.76) 1.00 0.71 0.88 0.99 0.71 0.71 0.71 0.71 0.71 0.71 0.71 0.71 0.71 0.71 0.71 LOAD EPOCH 1215 s node/492 8.40/15 (0.56) 0.99 0.46 0.46 0.46 0.46 0.46 0.46 0.46 0.66 0.46 0.66 0.46 0.99 0.46 0.46 LOAD EPOCH 1215 s node/29 11.57/15 (0.77) 0.75 0.75 0.75 0.77 0.77 0.77 0.77 0.78 0.83 0.77 0.78 0.78 0.77 0.78 0.75 LOAD EPOCH 1215 s node/413 9.97/15 (0.66) 1.00 0.60 0.60 0.60 0.78 0.60 0.60 0.60 0.60 0.60 0.60 0.60 0.60 0.60 0.99 LOAD EPOCH 1215 s node/19 11.77/15 (0.78) 0.76 0.74 0.75 0.77 0.77 0.78 0.78 0.79 0.78 1.00 0.78 0.78 0.78 0.78 0.75 LOAD EPOCH 1215 s node/432 9.12/15 (0.61) 0.57 0.57 0.75 0.57 0.57 0.57 0.57 0.57 0.57 0.57 0.57 0.57 0.57 0.57 0.99 LOAD EPOCH 1215 s node/193 14.55/15 (0.97) 0.96 0.96 0.96 0.96 0.96 1.00 0.96 0.96 0.99 1.00 0.96 1.00 0.96 1.00 0.96 LOAD EPOCH 1215 s node/236 13.43/15 (0.90) 0.99 0.74 1.00 0.89 0.89 0.73 0.89 0.89 0.89 1.00 0.89 0.89 0.89 1.00 0.89 LOAD EPOCH 1215 s node/487 9.22/15 (0.61) 0.47 0.47 0.99 0.66 0.47 0.99 0.47 0.47 0.47 0.47 0.66 0.66 0.99 0.47 0.47 LOAD EPOCH 1215 s node/351 11.99/15 (0.80) 0.99 0.70 0.88 0.71 0.99 0.73 0.88 0.70 0.71 0.71 0.71 0.71 0.88 0.99 0.71 LOAD EPOCH 1215 s node/148 14.95/15 (1.00) 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 LOAD EPOCH 1215 s node/260 13.61/15 (0.91) 0.85 1.00 0.99 0.85 0.85 0.85 1.00 0.99 0.85 1.00 0.85 0.85 0.85 0.85 0.99 LOAD EPOCH 1215 s node/18 11.56/15 (0.77) 0.75 0.75 0.75 0.78 0.78 0.78 0.78 0.78 0.78 0.78 0.78 0.78 0.78 0.78 0.75 LOAD EPOCH 1215 s node/424 9.32/15 (0.62) 0.58 0.58 0.77 0.58 0.99 0.58 0.58 0.58 0.58 0.58 0.58 0.58 0.58 0.58 0.58 LOAD EPOCH 1215 s node/65 14.95/15 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 ``` Memory at the end: [boisvert@miralac1 biosal-tests]$ grep MEMORY test-mira-512x16-1.output| tail MEMORY 2125 s node/138 13711466496 bytes MEMORY 2125 s node/339 13718646784 bytes MEMORY 2125 s node/163 12998049792 bytes MEMORY 2125 s node/474 14039306240 bytes MEMORY 2125 s node/187 13965811712 bytes MEMORY 2125 s node/450 13925138432 bytes MEMORY 2125 s node/6 14310481920 bytes MEMORY 2125 s node/498 14290821120 bytes MEMORY 2125 s node/59 14583406592 bytes MEMORY 2125 s node/506 13519155200 bytes