Pin workers to more than one accelerator

[WardLT]

Is your feature request related to a problem? Please describe. Parsl currently supports either pinning each worker to a single accelerator or no pinning at all. I want to pin each worker to more than one accelerator

Describe the solution you'd like Parsl have settings which support configuring more than one accelerator per GPU.

Describe alternatives you've considered Re-implementing pinning in my own Apps

Additional context

• @cms21 requested this feature
• it might be worth adding >1 worker per accelerator while we're at it

[WardLT]

I figure there are two routes:

1. implicit. Let the executor infer workers:GPU ratio if the user sets the max_workers and assume 1:1 otherwise
2. Explicit. Add a new option which defines the ratio

I favor the first, for parsimony, even though I normally shun implicit configs. We're not using anything about "max workers" being changed from the default yet

Any other thoughts?

[WardLT]

A third route could be to allow manually specifying worker->accelerator mapping

[benclifford]

the work queue/task vine executors let you specify, per-task, how many GPUs a task will get. This is a bit different from htex's symmetric approach that treats every task/worker as equal in resource demand, and is probably not the right way to go for this with htex, without also making htex treat all other resources as per-task.

[WardLT]

I agree. Having resource requirements at the task level would be advantageous. Could we file that as a longer term issue? That would require some major refactoring of the executor.

Accelerators are a good example. We'd have to kill the worker process between calls if the cuda visible devices changes. For many apps, that environment variable is read once on start. Htex doesn't support that kind of worker revivification yet

[benclifford]

I'm unclear that this should be a long term goal of htex at all: wq and task vine support this already, so there would have to be some decent justification for not using those (that further isn't "let's fix [the problem] in task vine")

[WardLT]

It sounds like we're in agreement about keeping htex with symmetric workers for now. I also thought that there wasn't much sense to making htex do something which wq already does.

In that case, what say you about how we should specify the GPU:worker ratio?

[benclifford]

I don't have particularly strong feelings about how this ratio is specified at the moment.

[cms21]

Hey all, basic question, when parsl "pins" an accelerator to a worker what does that mean? My understanding is that available_accelerators is used 1. to determine the number of workers to start on a node and 2. to set the appropriate environment variable. Is there anything else it's doing?

[WardLT]

Great question: the available accelerator list is used in a few steps:

1. as a limit to the number of workers per node in the executor: https://github.com/Parsl/parsl/blob/master/parsl/executors/high_throughput/executor.py#L269
2. as a limit for the limits the number of workers in the Manager: https://github.com/Parsl/parsl/blob/master/parsl/executors/high_throughput/process_worker_pool.py#L216
3. as a source of accelerator names when starting the workers: https://github.com/Parsl/parsl/blob/master/parsl/executors/high_throughput/process_worker_pool.py#L435

We'd have to update each of those lines break the assumption of 1 GPU:1 Worker. My vote would be to not use the accelerate list to to control the worker count in 1. That means the manager will receive a maximum number of workers equal to whatever the user provided in configuration, and can plan a mapping of worker->accelerator accordingly.

[cms21]

So just to clarify, if I were running on Polaris, say and set available_accelerators=["0,1", "2,3"], what would happen?

[WardLT]

It would create two workers and pin one to "0,1" and the second to "2,3." You know, that solves this whole issue.

Really, our problem is not whether Parsl can do something besides 1:1 worker:accelerator but that the documentation is wrong or its not easy to do it from the

@cms21 , do you happen to know if other accelerator runtimes (e.g., OneAPI) separate worker names by commas in their associated environment variables?