dzhang50
commented
8 years ago I gathered some data and created a graph to show you what I mean. This is very preliminary and rough data, the dynamic mesh network implementation is not good and is also modeling an extremely basic (read: low performance) mesh network. Also I was wrong about scaling for fixed latency. It turns out changing the static hop latency does greatly change absolute performance in cycles, but it affects performance by roughly the same factor regardless of the number of cores so the net result is that scaling relative to 1 thread isn't affected by static hop latency.
However, with a dynamic uncore model, it behaves like a low fixed latency model for small thread counts (low network contention) and more like a high fixed latency model for higher thread counts (high network contention). I suspect that my preliminary implementation of a dynamic uncore model is not good and much more testing is required, but it serves to demonstrate the point.
The graph below shows speedup relative to an infinitely fast static network model running BFS on 1 thread. The simulated microarchitecture has 64 OOO weave-type cores @ 2.5GHz, 2MB/core L3, and has a 6-channel DDR4 model that I added (don't really change the results vs. DDR3-1333-CL10 stock DDR weave-based model). Each line represents a specific network implementation: N-staticNet means a N-cycle per hop static network using the released ZSim network code. 3-dynNet means a 3-cycle per hop dynamic network using my custom initial dynamic mesh network implementation.
gaomy3832
s5z
shani-re
nirdavid
I'm still trying to solve my issues with ZSim scaling way too optimistically on irregular graph algorithms (e.g. BFS on Galois 2.2.1 with rmat4 input). Unfortunately, tweaking sim.phaseLength didn't help too much. I've identified the uncore network as a very likely reason for this issue: the statically-timed Network::getRTT() function assumes zero network contention. I was previously testing with 3-cycle total hop latencies (2 cycles for the router, 1 cycle for the actual hop like in your ZSim ISCA 2013 paper). Changing the network latencies (e.g. sweeping the router hop latency from 1 to 16) produces drastically different scaling results (2x+ cycle difference at 64 cores), as does my preliminary mesh network model that uses the CycleQueue in the OOO core model to model back-pressuring virtual channels (since getRTT() results are calculated once and cached, I also modified coherence_controls.cpp to call getRTT() for each network access). Obviously actual 64-core Westmere-class machines don't exist, but this correlates much better with real machines even on lower core counts (testing on 10 core 20 thread E5-2680 v2).
Unfortunately, it seems like in order for me to build a truly accurate mesh network model, I'll need to design a weave-based model. The weave code is very complex and is probably the part of ZSim that I understand the least. Does your research group already have an internal uncore model that's more detailed than the current getRTT() function? The ISCA 2013 ZSim paper mentions a mesh network timing model, and the MICRO 2015 SWARM paper mentions a more detailed mesh network with X-Y routing. If so, would it be possible to release it? If not, do you have any suggestions/tips on the proper way I can build one? I can probably emulate a mesh network with contention by just increasing the static latency per hop (clearly the easiest thing to do), but I suspect my advisor wouldn't like it if I told him I was doing this.