More clearly separate module- and runtime-oriented task private information.

[gbtitus]

Some of the information we need to store privately on behalf of each task is Chapel-oriented, having to do with language semantics or implementation, some has to do with the needs of the runtime, and some straddles both worlds. The full requested sublocale (to be) introduced with the KNL locale model work in PR #5555 is an example of information related to a locale model implementation, which the runtime doesn't need for its own purposes and indeed cannot directly interpret. The isExecuteOn flag is an example of purely runtime-oriented information. And the task's serial state is information that spans the boundary -- it is needed by both module code and the runtime.

Currently the runtime tasking layers, with help from the comm and other layers, define the entirety of the task private data record. Because of this, PR #5555 was forced into a bit of implementation clumsiness. The module code needed the task private data to store a "full" sublocale encoding that included both architectural and non-architectural information, such that when chpl_task_getSubloc() was called later it could return a value which combined the non-architectural information from the original requested sublocale specified when the task was created, plus the actual current architectural sublocale (NUMA domain for example) on which the task was executing at the time of the getSubloc() call. The runtime thus had to be able to split an architectural sublocale value out of a passed-in requested sublocale when creating the task, and merge an architectural sublocale value into a requested sublocale to be returned by getSubloc(), but without knowing what the requested sublocale actually looked like. This in turn required the locale model module code to export C-callable functions which the runtime could call to do these two operations. Worse yet, only the KNL locale model actually did anything in these two functions; the flat and numa locale model versions of them just passed back whatever they were given.

A better solution would be to define a new record type for task-private information that the module code needs to be able to interpret, and then the runtime could just include this in its task-private data struct. Ideally this record would be defined by the module code but from an implementation point of view that is impossible because the runtime is built first, before any Chapel code. So this would need to be defined as a C struct in the appropriate runtime/include/localeModels/*/chpl-locale-model.h file and referenced as an extern record type by the module code that creates and manages tasks. This record could be passed from the module functions that interface with the runtime tasking layer to the runtime functions that do the actual work, and returned by a new retrieval function in the runtime. This new type should contain a pair of sub-structs, one containing strictly module-oriented information which the runtime would not look inside of and another containing information shared between module code and the runtime which both need to be able to see. In the example of the KNL locale model, the non-architectural component(s) of the requested sublocale would go in the former and the task serial state would go in the latter. The runtime chpl_task_getSubloc() function could then return strictly architectural sublocale values, and the KNL locale model module code could create the "full" sublocale values it needed by combining the non-architectural sublocale information it had passed in the information record when creating the task, with the current architectural sublocal from getSubloc().

Note: this only solves the problem for locale model oriented info. But so far that's all we need, and if there were other parts of the modules or Chapel language that could use this it seems like we could add defining code in the runtime to handle those also.

The following runtime functions seem like they'd get a simpler interface as a result of this work:

• chpl_task_addToTaskList()

• chpl_task_taskCallFTable()

• chpl_task_startMovedTask()

The following would be new:

• chpl_task_getPrivateData()

(There is a chpl_task_getPrvData() but it's for internal runtime use; we might need to rename it for clarity.)

The following would probably no longer be needed, since Chapel code could achieve the same effect by calling chpl_task_getPrivateData() and plucking the information out of the record:

• chpl_task_getRequestedSubloc()

• chpl_task_getSerial()

• chpl_task_setSerial()

Current Status

I still need to do some more investigation to see if this idea is actually practical and won't hurt performance, although the latter seems unlikely.

[gbtitus]

A number of PRs have improved matters since this Issue was written, though unfortunately I didn't record related PRs here as they were merged. I think there is more that could be done, for example see https://github.com/Cray/chapel-private/issues/957. I'm going to leave this open for the group to decide what to do with, but am removing myself as the assignee.