dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
45 stars 107 forks source link

Implement a resource awareness mechanism to handle workflows with hybrid resource needs #11569

Open khurtado opened 1 year ago

khurtado commented 1 year ago

Impact of the new feature This feature would allows us to have better awareness of the type of resources hosted by each site: E.g.: GPUs, CPUs with Intel/AMD architecture, ARM, PPC architectures, etc.

Is your feature request related to a problem? Please describe. This issue is related to assignment of workflows to sites with heterogeneous resources. For example, let's assume we have a task chain workflow where:

If we run Task 1 runs in a site with amd64 architecture but didn't have any arm architecture, then Task 2 can potentially get stuck due to data-driven requirements (outputs waiting on Site 1, so we will match to Site 1 for data transfer efficiency), so we need to be aware of this by the time we are doing the workflow assignment.

Describe the solution you'd like This feature would need to be implemented somewhere, like Reqmgr2 or in a component performing the workflow assignment (Unified at this point). The solution design still needs to be evaluated

Describe alternatives you've considered none

khurtado commented 12 months ago

@amaltaro I can work on this one!

khurtado commented 10 months ago

Back to TODO, as this is tightly coupled with workflow assignment and this could be covered by this service. Further discussion needed.