Expose targetLocales indices to the user?

[vasslitvinov]

This issue proposes an alternative to #11930. That issue brings up the problem that hasSingleLocalSubdomain returns a param value and that there is localSubdomain. These don't make sense in the event that, say, a Block array is created with targetLocales that uses the same locale twice.

Currently distributions like Block store the locales that the indices are to be distributed over in an array targetLocales. "Local descriptor" classes (in DSI slang) are stored in arrays of the same shape as targetLocales. When some computation needs to access a local descriptor, it needs to perform a translation from here to the index into targetLocales. This happens, for example, for queries like localSubdomains(). This may be complicated because there may be multiple indices into targetLocales in the case of oversubscription.

This issue proposes to expose the indices into targetLocales to the user, for example via an iterator. This will enable:

• easy handling of certain operations in the case of oversubscriptions,
• determining the locale that is the left/right/... neighbor for the current computation.

In particular, given an index into targetLocales, a Block array really does have only a single subdomain for that index, so localSubdomain would accept that index as an argument.

[vasslitvinov]

Related: #11930

[vasslitvinov]

The idea of exposing the indices into targetLocales could be seen as an instance of the more general query "what is my position within the enclosing forall-loop?" For example, if the computation is in the body of a forall-loop whose parallel iterator has a coforall over locales then coforall over tasks, it may be handy to know the current task's index within each coforall.

[mppf]

The idea of exposing the indices into targetLocales could be seen as an instance of the more general query "what is my position within the enclosing forall-loop?" For example, if the computation is in the body of a forall-loop whose parallel iterator has a coforall over locales then coforall over tasks, it may be handy to know the current task's index within each coforall.

This is related to issues #14405 and #16405.

[mppf]

One thing I am curious about is if the distributions that motivated localSubdomains could be expressed as another distribution with oversubscription in targetLocales. For example, to have a Block Cyclic distribution of block size 2, we could imagine having a targetLocales array that consists of [ Locales[0], Locales[0], Locales[1], Locales[1], ...] i.e. lists each locale twice.

Now, it probably does not make sense to actually store such a targetLocales array, but could we view something like BlockCyclic as a Cyclic distribution that optimizes the storage of the targetLocales array and a bit of the other elements for this case?

In that event, we wouldn't have localSubdomains. We would have:

• iter localIndicesIntoTargetLocales (better name TBD) that yields indices into targetLocales where targetLocales[idx] == here

• localSubdomain(indexIntoTargetLocales)

  and these would be sufficient even for BlockCyclic. One potential downside of this approach is that BlockCyclic need to have a different idea of "target locales" that you provide to the initializer. So there would be two different things we might want to call "target locales". We could say that it is canonicalized/normalized or use a different term for the expanded one from the one the user provides.

[e-kayrakli]

Now, it probably does not make sense to actually store such a targetLocales array, but could we view something like BlockCyclic as a Cyclic distribution that optimizes the storage of the targetLocales array and a bit of the other elements for this case?

In that event, we wouldn't have localSubdomains.

I am not following this. Technically, I can write funky targetLocales arrays for Block/BlockCyclic/Cyclic and make them effectively be distributed like any other in that group. We can also write targetLocales that makes these distributions map to single locale.

In general, I think we should expose indices into targetLocales to the user somehow. But I wonder if we can make it via a higher-level concept somehow. Where I am imagining distribution objects can have a "locale map" object of sorts that can be used to query such things as locale-to-location and location-to-locale mappings. For some reason, having multiple methods/fields to query nuts-and-bolts of these mapping feel cumbersome to me.

[bradcray]

One thing I am curious about is if the distributions that motivated localSubdomains could be expressed as another distribution with oversubscription in targetLocales.

I think they can, but I don't think they should be. BlockCyclic can certainly be thought of as a Block distribution in which targetLocales are repeated n/blockSize times; or as a Cyclic distribution in which the target locales are repeated blockSize times consecutively as you indicate. But I don't think it's attractive to either the user or the implementer of BlockCyclic to think of it that way. And I don't think we should say that the only distributions one is permitted to write in Chapel are those in which only a single subdomain can be stored by each locale. Basically, I don't think simplifying this interface in this way is worth the "squint at it just right to make it fit this pattern" impact it would have on users and domain map authors. If we wanted to simplify the interface, I think the approach to take would be to only support the iterator form (which would also have the benefit of permitting a locale to say "tell me all the subdomains I own" in the case of oversubscription).