search

illinois-ceesd / mirgecom

MIRGE-Com is the workhorse simulation application for the Center for Exascale-Enabled Scramjet Design at the University of Illinois.
Other
12 stars 19 forks source link

Ballooning parallel times #49

Open MTCam opened 4 years ago

MTCam commented 4 years ago

Ballooning setup times vs. number of elements have been observed on Quartz. Looks like these are associated with the mesh distribution (i.e. mesh_dist.send_mesh_parts ) NOT partitioning with metis.

Serial I/O rates seem ok - I'm checking (with LC) on what to expect from NFS on this particular FS quartz_serial_nfs_io_rate

Parallel I/O rates do not scale as expected. Our viz files write in roughly constant time at (IO BW HERE). I/O also does not seem to be a parallel bottleneck (yet). Here are initial parallel timings from Quartz (CPU only), gotten by strong scaling the Y0 case (running only 5 steps). Y0ParallelTimes_1 5 steps may not be enough to see the actual scaling of advancement, due to (code generation?) on the first call to constructs using kernels.

inducer commented 4 years ago
inducer commented 4 years ago
MTCam commented 4 years ago

Here's the raw write speeds gathered with LukeO's code here running on Quartz on the shared file space where the previous tests were run. rates.pdf

And here is the one from running the same benchmark on the parallel filesystem, but from Lassen this time due to Quartz DAT. I'll grab the same timing data for that filesystem and start running exclusively there from the parallel filesystem from now on.

lassen_io_rates_scratch.pdf