for_each in pipeline step should support limits on containers

[Priyanka-Chatterjee-2000]

Hi , I was trying to create multiple (100) GCP storage buckets using flowpipe, I could think of two ways to do it:

Using for_each: With this 100 container spun up at an instant in parallel clogging up all my system resources and eventually leading to a shutdown. So to tackle this issue I tried the second approach.

Using loop: We can make synchronous repeat calls to a step until a certain condition evaluates to true . As per the documentation, I should be able to overwrite the arguments in any kind of step, the example also shows the same about a http step. However in a pipeline step, I cannot seem to overwrite the "args" argument and it complains about this error:

Error: Failed to decode mod:
Unsupported argument: An argument named "args" is not expected here.
(/Users/priyankachatterjee/workspace/turbot/pipeling-scale-testing/scripts/gcp/create_multiple_storage_buckets.fp:45,7-11) 

:link: Link to my code

Suggestions from @e-gineer : We should support both. I think for_each is the better approach … and for that to work we should support limits on containers. I think we should have two limits we respect:

• A max_parallel attribute on the for_each that can be set to control parallelism for that specific step.
• A global max_parallel_containers field that works across all containers for all steps at once. This should have a default value to prevent floods - eg 100

[vhadianto]

Will be done as part of v0.3.0

Current beta: https://github.com/turbot/flowpipe/releases/tag/v0.3.0-beta.202402130018 supports this feature.

[vhadianto]

This is now supported as part of https://flowpipe.io/docs/flowpipe-hcl/step#common-step-arguments