Should shmem_wait_until take a local or symmetric address?

[nspark]

PR #276 clarified the distinction between local addresses and symmetric addresses. It also updated the API listings to specify whether they expect local or symmetric addresses to remote data objects.

As of 1.5rc1, shmem_wait_until expects ivar to be a "local address of a remotely accessible data object," and shmem_ptr returns "a local address to a remotely accessible data object." This allows the following behavior:

const int mype = shmem_my_pe();
const int npes = shmem_n_pes();
long *sym_flag = shmem_calloc(1, sizeof(long));
long *lcl_flag = shmem_ptr(sym_flag, mype);
assert(lcl_flag != sym_flag);

if (mype == 0) {
  shmem_wait_until(sym_flag, SHMEM_CMP_EQ, npes - 1); // clearly allowed
  shmem_wait_until(lcl_flag, SHMEM_CMP_EQ, npes - 1); // questionably allowed
}
else {
  shmem_atomic_add(sym_flag, 1, 0);
}

This example is correct (per 1.5rc1) even if shmem_ptr is evaluated with respect to a PE other than the calling PE; e.g., if two PEs are on the same node and the implementation allows load/store access between PEs via shmem_ptr, can one PE call shmem_wait_until on the local address to the other PE's object? (My vote: no.)

Do we want to prohibit such behavior? (My vote: yes.)

Proposed fix: Require shmem_wait_until and friends to require ivar[s] to be a "symmetric address of a remotely accessible data object."

[nspark]

Tagging the Synchronization, Ordering, and Locking Section Committee: @nspark @naveen-rn @BryantLam @akhillanger

[naveen-rn]

can one PE call shmem_wait_until on the local address to the other PE's object?

NO

Do we want to prohibit such behavior?

YES

The proposed change looks fine with me.

[BryantLam]

Yes, I agree with this change.

Require shmem_wait_until and friends ...

I assume this statement means every function in the point-to-point synchronization section.

[jdinan]

👍

[naveen-rn]

Can we close this issue as the above change is merged into the section committee branch?