clear separation of Dart/DASH and MPI

[dhinf]

Now, we are implementing a Gaspi runtime in DART, some MPI specific stuff pops up. Like using MPI calls in TestPrinter.h. Also, the cmake options for shared and dynamic windows should have MPI as part of the name, to mark them as MPI specific.

[fmoessbauer]

Are there any other locations (except the TestPrinter) where MPI is used directly? The TestPrinter itself is a bit special as it coordinates the execution of the tests and collects and unifies the results. Hence it is totally valid to use MPI / whatever there directly as neither DASH nor DART can be used there.

Technically this is because DASH is initialized and teared-down after each testsuite.

The point is that you have to specialize the TestPrinter for each particular interface. This is currently done for SHMEM and MPI. While you do not have a self-containing implementation using GASPI, just use the SHMEM implementation as a fallback. This should work, but the results are not collected and the overall result might be misleading.

[devreal]

More interesting to me, what are the DART functionalities that are hard/impossible to implement using GASPI? I'm thinking about dart_send and the collective operations.

Just out of curiosity (and potentially to ease the transition): isn't it possible to run GASPI and MPI side-by-side? So you could leave the TestPrinter as is and still use GASPI as the DART frontend?

[dhinf]

It is possible to run both gaspi and mpi. The point is, that MPI is only a runtime and should by accessed via the dart interface only. If this is not possible, our concept doesn't fit anymore (just my opinion). Maybe i'm wrong here :) .

@fmoessbauer: TestPrinter is the only location we found at the moment @devreal: their are some things gaspi is not capable of, like remote completion for a put.

We try to run the test suite at the moment and it's possible we find other artefacts.

[devreal]

their are some things gaspi is not capable of, like remote completion for a put.

Interesting. We rely on remote completion quite significantly. Our memory model dictates that remote writes should be visible as soon as the operation is completed, so that the next read (on the same unit) returns the written value. Is sequential ordering of accesses from the same process guaranteed? At which point is remote completion guaranteed?

I wouldn't worry too much about the TestPrinter for now as this is just support infrastructure and not needed to use DASH in actual applications.

[dhinf]

Is sequential ordering of accesses from the same process guaranteed?

No

At which point is remote completion guaranteed?

Good question. Without a special demon managing the communication you can't guarantee remote completion. With get it is no problem, but put/write you can only guarantee local completion.

[devreal]

Looks like you need to use gaspi_wait to guarantee sequential ordering. There seems to have been some confusion among the GASPI peeps about the term "remote completion": https://github.com/cc-hpc-itwm/GPI-2/issues/22

We should make sure that a barrier is sufficient to ensure that values written before the barrier are visible to all processes after the barrier. That is still somewhat weaker than our current guarantees (as famously defined in our unofficial memory model ^^) but maybe we can use the opportunity to define the memory model and exploit the weaker semantics in the MPI backend as well. It would also be interesting to see whether we can make use of the notification scheme in GASPI for p2p signaling.

[pascalj]

gaspi_wait only guarantees that operations have been processed by the network. The write operations may still be pending on the remote host. See page 65 of the standard:

User advice: Return value GASPI_SUCCESS means, that the data of all posted write requests in this queue is in transfer to the remote side. It does not mean, that the data has arrived at the remote side. However, write accesses to the local source location will not affect the data that is placed in the remote target location.

The GASPI people said the way to guarantee remote completion is to send a notification back to the sender.

[dhinf]

The GASPI people said the way to guarantee remote completion is to send a notification back to the sender.

And to do that, you need to use a demon.

[devreal]

The GASPI people said the way to guarantee remote completion is to send a notification back to the sender.

That sounds horrible :( IMO, a daemon is not an option. We'd have to implement our own all-to-all notification scheme to signal remote completion on the global scale. Am I missing something?

[dhinf]

IMO, a daemon is not an option.

definitely Like i said before. Local completion isn't a problem. remote works for get/read. But put/write needs some kind of notification that the values are written. At the moment, i have no idea how to trigger the remote unit to send a notification.

[pascalj]

And to do that, you need to use a demon.

Yes! I didn't want to say otherwise.

We'd have to implement our own all-to-all notification scheme to signal remote completion on the global scale. Am I missing something?

You're right. I think in principle it wouldn't be that difficult to build, but it kind of defeats the purpose of GASPI's design/model.

@dhinf Out of curiosity: do you have any plans regarding that or will you cross the bridge when you get there?

[dhinf]

@pascalj i will convince users that it is best practice to use get/read instead of put/write. problem solved ;)

Seriously, we will try to find a solution. How long it takes, i can't estimate at the moment.