Support for querying the 'localSubdomain' and 'localSubdomains' of other locales

[LouisJenkinsCS]

It would be pretty cool if it was possible to call localSubdomain(otherLoc) to obtain the localSubdomain, or localSubdomains(otherLoc) to obtain the localSubdomains of other locales. This can save a round-trip for best-case scenarios where it can be computed locally by a remote locale, or even better, the subdomains can be cached on all privatized domains for distributions so that it will always be local. This would be a pretty nice and easy solution.

proc localSubdomain(targetLoc = here) {...}
iter localSubdomains(targetLoc = here, ...) {...}

[bradcray]

I worry about caching remote sublocales in general due to the memory consumption it would require for programs with lots of domains designed for scalability. But happily, this policy choice could be made by the implementer of a distribution in terms of how they want to implement the localSubdomain*() queries (i.e., whether it's communicated, computed, or cached isn't a decision that the language or interface would impose; a domain map author could do any of these things, or a combination of them).

[bradcray]

@LouisJenkinsCS: Just as a heads-up, I've been working on prototyping this for "Block" this week and have a rough draft. Were there specific domain maps or distributions that you were most interested in seeing it on, to help prioritize? Thanks.

[LouisJenkinsCS]

Definitely for Cyclic as well, perhaps BlockCyclic too. I'm assuming that for Cyclic it would be a lot easier considering it could just compute it based on the stride. The two highest-priority distributions definitely are Block and Cyclic since those are the most used.

[bradcray]

The ones you use the most, or the ones you conjecture others use the most? :) (I'll do them all eventually, just trying to get you a solution that's useful to you in short order if possible).

[LouisJenkinsCS]

To backup my conjecture, there are 269 files which instantiate Block, 73 files which instantiate Cyclic, 28 files which instantiate BlockCyclic, 15 files which instantiate Stencil. Hence, Block > Cyclic > BlockCyclic > All else. Q.E.D

[bradcray]

I'm not saying I disagree with your conjecture (though I wouldn't use our test system as proof since it's only indicative of where we've invested effort, not what users have been using, so it's a bit of a self-fulfilling prophesy). But here's what I'm really trying to ask: In the case(s) where you ran into this, causing you to open the issue what distribution(s) were you using? I'm planning to implement the interface for all domain maps before closing this issue, but since I can only do one at a time, wanted to prioritize cases that would help your code and/or let you remove any workarounds you're using currently. Thanks.

[LouisJenkinsCS]

Gotcha, my 'backing up my conjecture' was meant to be lighthearted. To reiterate, my most used distribution BlockDist and second most is CyclicDist. Thank you for doing this.

[bradcray]

@LouisJenkinsCS: Over on PR #11979, I've got this implemented for arrays and domains dmapped using default, Block, Stencil, Cyclic, Replicated, and Hashed distributions. I believe Block-Cyclic and Sparse Block are the two cases that don't support it at this point.