Enable setting pool in the expand method of a dynamic task

[filipe-alves-itech]

Description

Allow pools to be set dynamically for each dynamic mapped task index via the expand method as opposed to being limited to the current behaviour that only allows a static definition for all mapped tasks via the partial method.

Use case/motivation

Pools are very useful for having control over resource usage on the backend systems that can oftentimes be overwhelmed by concurrent tasks triggered by Airflow. With dynamic mapped tasks, we can have a scenario where different mapped instances may trigger work in different backend servers and having the ability to set different pools comes in handy allowing control over the number of running tasks for each backend server separately. Dynamic tasks were a game changer allowing to dynamically define parts of the DAG, making it more like a first-class citizen that benefits from the other established Airflow features like pools is another step forward.

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Not sure if that would be easy, but I think there is nothing fundamentally difficult here (I believe it should be quite possible.

[potiuk]

@uranusjr - do you see any fundamental problems with it ? If thera are no fundamental limits (I can't think of any) it could be even fairly easy - and we could mark it as "good first issue" maybe ?

[uranusjr]

I think this is possible in theory, but before this is possible we’d need to first make pool available via XCom without task mapping involved.