search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.47k stars 1.64k forks source link

Artificial Concurrency Delay for the Map Function #15752

Open majo-aqfer opened 1 month ago

majo-aqfer commented 1 month ago

Describe the current behavior

The current map functionality is very useful for triggering the same task with different input values simultaneously. However, when we utilize the map function, all tasks start executing at the same time without any delay between them. This concurrent execution can create conflicts during the run, leading to issues such as resource contention or race conditions.

Describe the proposed behavior

To enhance the reliability and efficiency of the map function, it would be beneficial to implement a feature that allows users to control the execution delay between tasks which is running from map. By doing so, we can mitigate conflicts during task runs and ensure a smoother workflow overall.

Example Use

No response

Additional context

No response

zzstoatzz commented 1 month ago

hi @majo-aqfer - thank you for the issue!

would you be able to fill out the Example Use section so we can get an idea of what DX you're interested in?