garlick
commented
8 years ago Quick opal test:
$ time srun -N8 ./flux broker flux getattr size
8
real 0m1.866s
user 0m0.016s
sys 0m0.030s
$ time srun -N16 ./flux broker flux getattr size
16
real 0m2.568s
user 0m0.026s
sys 0m0.043s
$ time srun -N32 ./flux broker flux getattr size
32
real 0m2.859s
user 0m0.039s
sys 0m0.079s
$ time srun -N64 ./flux broker flux getattr size
64
real 0m11.841s
user 0m0.058s
sys 0m0.098s
$ time srun -N128 ./flux broker flux getattr size
128
real 0m7.147s
user 0m0.094s
sys 0m0.194s
$ time srun -N128 -n256 ./flux broker flux getattr size
256
real 0m9.238s
user 0m0.114s
sys 0m0.187s
$ time srun -N128 -n512 ./flux broker flux getattr size
512
real 0m16.258s
user 0m0.127s
sys 0m0.220s
$ time srun -N128 -n1024 ./flux broker flux getattr size
1024
real 0m39.199s
user 0m0.175s
sys 0m0.326s(got core dump at size 2048 - see flux-framework/flux-core#683)
Baseline with mpi_hello compiled with Intel MPI which uses PMI:
$ time srun -N 8 ./hello
0: completed MPI_Init in 0.163s. There are 8 tasks
0: completed first barrier in 0.000s
0: completed MPI_Finalize in 1.004s
real 0m2.443s
user 0m0.033s
sys 0m0.047s
$ time srun -N 16 ./hello
0: completed MPI_Init in 0.671s. There are 16 tasks
0: completed first barrier in 0.000s
0: completed MPI_Finalize in 1.004s
real 0m3.077s
user 0m0.047s
sys 0m0.081s
$ time srun -N 32 ./hello
0: completed MPI_Init in 0.973s. There are 32 tasks
0: completed first barrier in 0.000s
0: completed MPI_Finalize in 1.004s
real 0m3.219s
user 0m0.107s
sys 0m0.130s
$ time srun -N 64 ./hello
0: completed MPI_Init in 1.376s. There are 64 tasks
0: completed first barrier in 0.000s
0: completed MPI_Finalize in 2.004s
real 0m3.685s
user 0m0.182s
sys 0m0.193s
$ time srun -N 128 ./hello
0: completed MPI_Init in 2.320s. There are 128 tasks
0: completed first barrier in 0.000s
0: completed MPI_Finalize in 1.004s
real 0m4.962s
user 0m0.265s
sys 0m0.367s
$ time srun -N 128 -n 256 ./hello
0: completed MPI_Init in 1.404s. There are 256 tasks
0: completed first barrier in 0.000s
0: completed MPI_Finalize in 2.005s
real 0m4.205s
user 0m0.351s
sys 0m0.440s
$ time srun -N 128 -n 512 ./hello
0: completed MPI_Init in 2.408s. There are 512 tasks
0: completed first barrier in 0.000s
0: completed MPI_Finalize in 2.007s
real 0m5.197s
user 0m0.396s
sys 0m0.757s
$ time srun -N 128 -n 1024 ./hello
0: completed MPI_Init in 4.480s. There are 1024 tasks
0: completed first barrier in 0.001s
0: completed MPI_Finalize in 2.043s
real 0m7.785s
user 0m0.470s
sys 0m0.984s
$ time srun -N 128 -n 2048 ./hello
0: completed MPI_Init in 7.954s. There are 2048 tasks
0: completed first barrier in 0.001s
0: completed MPI_Finalize in 4.245s
real 0m13.462s
user 0m0.732s
sys 0m1.564s
$ time srun -N 128 -n 4096 ./hello
0: completed MPI_Init in 17.283s. There are 4096 tasks
0: completed first barrier in 0.002s
0: completed MPI_Finalize in 8.338s
real 0m27.137s
user 0m1.431s
sys 0m3.099s
grondo
lipari
_Goals_
Determine Flux startup/teardown scalability. Find any bugs revealed at scale.
_Methodology_
Gather baseline SLURM job launch data, so SLURM overhead can be subtracted from Flux times. Use MPI program that bootstraps with PMI like Flux. This SLURM data may also be useful for comparison with Flux job launch data (not covered here). Get enough runs to extract valid statistics. Run at various scales, e.g. 1,2,4,8,12 scalable units (one task per node).
Instrument and time the following phases of Flux startup/teardown:
Run through same scales as baseline, and run enough to extract valid statistics. Log issues for all anomalies linked from #12
Extra credit: continue baseline and Flux scaling to one task per core.
Publish baseline and Flux statistics in this issue.
_resource requirements_
Ramp-up beginning on opal (1 SU, 192 nodes, 6912 cores) to full CTS-1 system.
_exit criteria_
srun -N2304 flux start /bin/true_Issues to watch out for_
flux module load -r allis called serially in rc1 and the converse in rc3, and seems not to scale well (more info needed)