Closed ronawho closed 4 years ago
Here are some runs from a Cray-CS that has similar per-node hardware to the main Cray-XC runs we did (36-core 2.1 GHz Broadwell CPUs with 128 GB RAM). My runs are with Chapel 1.20, intel 19.0.3.199, and cray-fftw 3.3.8.4 (so similar software as well):
nodes | ugni | gn-ib-lrg | gn-ib-fast |
---|---|---|---|
8 | 32.63s | 148.76s | 37.93s |
16 | 16.99s | 83.42s | 20.10s |
32 | 10.40s | 24.69s | 10.69s |
nodes | ugni | gn-ib-lrg | gn-ib-fast |
---|---|---|---|
16 | 162.79s | 526.58s | 204.00s |
32 | 88.04s | 295.76s | 103.86s |
gn-ib-large suffers, but gn-ib-fast performance is relatively close to ugni, which means that comm performance is more important than NUMA-affinity for this benchmark (which we knew already.)
(This matches what I was expecting and roughly tracks ISx ratios at 16 nodes.)
I don't think there's anything for you to do here. Improving IB performance is one of the things I'm working on for next release, and I expect this will fallout from this. This code is mostly doing bulk comm, and IB hardware is great at that, so we should be able to match Aries here. Note that this is 56 Gb FDR InfiniBand (newer hardware should be able to do even better.)
Nightly results are available here
For now, I recommend using CHPL_GASNET_SEGMENT=fast
for gasnet ibv.
There have been some recent improvements to InfiniBand performance (and I'm expecting we'll have a 1.24.1 release that includes these.)
Reminder that we have 2 modes we can run gasnet ibv with: segment large, and segment fast. Segment large dynamically registers memory so it gets good NUMA affinity, but comm performance historically suffered. Segment fast statically registers at startup so NUMA affinity will be wrong and startup is slow, but comm will be fast.
https://github.com/chapel-lang/chapel/pull/17405 improves the startup time and slightly improves NUMA affinity for segment fast. https://github.com/chapel-lang/chapel/pull/17505 significantly improves comm performance for segment large.
The nightly results are available here. On 03/18 we see slight FT improvements for gn-ibv-fast due to slightly better NUMA affinity and significant reductions in startup time. On 04/01 we see significant FT improvements for gn-ibv-large due to comm improvements.
I no longer have access to a machine that has similar per node hardware to an XC so I can't gather ugni comparisons, but here are some large vs fast segment scaling comparisons on that same nightly machine:
nodes | gn-ibv-large | gn-ibv-fast |
---|---|---|
8 | 34.76s | 36.02s |
16 | 19.29s | 20.86s |
32 | 11.24s | 10.51s |
64 | 5.82s | 5.84s |
nodes | gn-ibv-large | gn-ibv-fast |
---|---|---|
16 | 206.37s | 164.57s |
32 | 106.59s | 88.11s |
64 | 59.49s | 45.78s |
For size D the runtimes are about the same (large slightly ahead because of better NUMA affinity). Segment large lags with larger problem sizes (and this isn't surprising, there is more work to do here) but it is significantly better than before. I'd probably still recommend CHPL_GASNET_SEGMENT=fast
for this, startup times aren't bad anymore and you'll still get better performance at larger problem sizes.
Spawned off from https://github.com/npadmana/DistributedFFT/issues/48#issuecomment-541276144 --
I haven't run this on our local cluster yet (and I don't have enough nodes to do a real stress test), but it would be interesting to see how the code does under gasnet (for a cray but also for ib).
All our timings so far have been with ugni on Cray-XCs. It'd be interesting to gather some IB results too.
I can do some smaller-scale IB runs. For sure 32 nodes, possibly 64. I would bet performance will currently lag on IB by ~2x. That's something we're actively working on. IB hardware has good bulk comm performance, so hopefully within the next release or 2 IB performance should be on par.
edit:
More details and a better summary of gn-ibv perf is captured in https://github.com/chapel-lang/chapel/issues/14438