Atomic operation mix in DASH

[devreal]

As discussed at the F2F today, DASH currently allows to use an arbitrary mix of atomic operations on objects in the global address space. However, MPI in version 3.1 only allows for either the same operation or the same operation and no-op to be performed concurrently on the same memory location (see §11.2.1 on p. 404 in the standard). A proposal to the MPI forum would add an info key which_accumulate_ops to specify the mix of operations to be expected.

The current implementation of atomics in DASH is error-prone as it is easy to violate the rules of the MPI standard in its current form and hard to signal a restricted set of operations in case the new info key is being introduced. The latter will be needed to efficiently exploit hardware capabilities and may lead to performance issues with atomics because the MPI implementation cannot assume that it is safe to use hardware-based atomics.

My initial proposal for DASH was to incorporate the allowed atomic operations into the dash::Atomic type, e.g., dash::Array<dash::Atomic<T, dash::plus>> to signal this information to the underlying window and disable all operations in the interface except for addition (which would also allow us to enforce adherence to the single-operation rule).

As discussed at the F2F today, there are two main problems with this approach (my recap, please correct if wrong):

1) The global memory allocator knows nothing about the type it allocates memory for and it is a violation of the separation of concerns as the info key has to be passed to the segment allocation although it is a concern of the objects in the segment.

2) We use accumulate functions outside of dash::Atomic, e.g., in dash::transform.

I late realized that there is a function MPI_Win_set_info that allows to set info keys after the window has been created. Thus, we would be able to set this info key for a particular phase of the execution, e.g., at the beginning and the end of dash::transform. We still would need a way for the user to signal a restricted set of atomic operations for use with dash::Atomic, e.g., by modifying the setting of the segment once it has been created. The interface could look something like dash::set_atomic_ops(matrix, dash::plus, dash::compare_exchange). That, however, would still leave the dash::Atomic interface unrestricted and leave the door open for potential misuse.

[fuchsto]

Among the topics discussed in the DASH team summit this week, I found this problem the most intriguing. It is orthogonal to any common conceptual category of parallel programming. I mean, it cannot be fully described in terms of:

• value semantics
• memory access
• container semantics ... etc.

I think we need more fine-grained concepts of allocation? Perhaps we need to differentiate between "memory" and "MPI window"?

[devreal]

You're right, it's more like a state that can be changed that influences the way in which memory accesses are performed, not their semantics but potentially their performance. Given the fact that these hints can be set after the memory/window has been allocated we should probably decouple this issue from memory allocation and focus on the runtime aspect. In fact, there may be situations where we can guarantee a single op (such as dash::transform or if the user of dash::Atomic knows what he is doing). There is some resemblance to the memory ordering hints to std::atomic. Although those are relevant for correctness, the default is memory_order_seq_cst and if a user knows the intricate details and doesn't mind footguns he can relax the constraints and potentially gain performance. Maybe we can come up with something similar for dash::Atomic (or containers, since this is not operation specific)?