search

ComputationalRadiationPhysics / graybat

Graph Approach for Highly Generic Communication Schemes Based on Adaptive Topologies :satellite:
Other
8 stars 4 forks source link

issue #38 and vertex/edge communication interface #39

Closed erikzenker closed 9 years ago

erikzenker commented 9 years ago

Vertices provide communication interface

bussmann commented 9 years ago

In MPI the naming is different (scatter, gather, reduce), why the changes?

erikzenker commented 9 years ago

These communication methods are no collectives, but a connection of basic point to point communication methods. I did not want to make the impression that these are the same as MPI collectives. On the CommunicationPolicy level methods like scatter, gather and reduce exist and these are real collective in the MPI sense.

I still think about an approach how to control a communication method like gather/collect to be once collective and once a connection of basic point to point operations. It would be nice to have such kind of switch at runtime without a lot of hassle.

But since I am not finished with considerations I gave them different names, but still confusing I know.

ax3l commented 9 years ago

but a connection of basic point to point communication methods

can't I interpret those as a collective communication in a (virtual) communicator that only includes the connected points of all adjacent vertices?

bussmann commented 9 years ago

@ax3l: But one would still need to identify the connection graph.

ax3l commented 9 years ago

so a spread might be directed and only works from let's say a specific source in the graph to actually reach all nodes?

@erikzenker a tree example and two single-connected start networks would be nice.

bussmann commented 9 years ago

^^ would love to see these examples.

erikzenker commented 9 years ago

@ax3l your are right that you could create an (mpi ) communicator for a (sub )graph,but keep in mind that mpi collectives are blocking.

A spread is a non blocking send from a source vertex to all adjacent vertices taking the graph into accountc. A spread to all vertices needs a star graph were all vertices are connected to the source.

@ax3l so two trees connected by one edge?

ax3l commented 9 years ago

@erikzenker MPI-collectives are not necessarily blocking, MPI 3.0 introduces non-blocking variants for all blocking calls (and MPI 3.1 is already out). They are actually very useful and extremely needed.

Implementations that support all MPI 3.0 calls are MPICH 3.x and OpenMPI (statuses as of Nov 2013, OpenMPI hp: "Full MPI-3 standards conformance" in current releases) ... and of course all their derivatives as long as they keep their upstream changes up to date.

Btw: MPI 3.1 also adds support for non-blocking collective I/O calls :D (first features in mpich)

Long story short: non-blocking MPI collectives are super-awesome (such as IBarrier) and here to stay :)

trees: yes just connected by one edge, e.g., make two binary trees of each 3 levels (root, mid, leaves) and connect one of the mid-nodes with the other tree.

dbltree

ax3l commented 9 years ago

oh wait, now I made it too complicated (but also sexy: partially connected hyper-tree without connected root, njam). I was posting those two layouts earlier: simple (single) tree from above and a connected double star network:

# ascii art rulez, too

1   2   3              8   9   10
\   |   /               \   |   /
 \  |  /                 \  |  /
   \|/                     \|/ 
    4 --------------------- 11
   /|\                     /|\ 
 /  |  \                 /  |  \ 
/   |   \               /   |   \
5   6    7             12  13   14

of course, connected hyper-trees are very common in HPC hardware: both for interconnects to the connected I/O or even for the cluster itself when money for a full 3D torus was not available or the system was upgraded over several islands over time.

erikzenker commented 9 years ago

@ax3l thx for that hint ...so in the case of an mpi communication policy i could use these non blocking collectives. And maybe a template argument to switch between blocking or non blocking would be nice.i need to test it and have a look if there sre some fallstricke in connection with graybat.

Okax we have a network topologie, now need an applikation and its communication pattern that should be mapped.do you have something in mind?

ax3l commented 9 years ago

sounds great! the double star could be a very small GoL with two islands. In the real world (TM) this would be something like 6 GPUs on one infiniband-card per node.

the tree network could be just the underlying physical network of a cluster where one actually needs a connection of 6 neighbors (via routing in the tree) in a virtual cartesian grid of nodes. so every cartesian application, e.g., GoL would be possible.