search

Azure / AzOps

AzOps is a PowerShell module which deploys (Push) ARM Resource Templates & Bicep files at all Azure scope levels and exports (Pull) ARM resource hierarchy.
https://aka.ms/AzOps
MIT License
384 stars 163 forks source link

Parallel Deployments with Multiple Parameter Files fail with file in use #868

Closed DennisSchouwenaars closed 6 months ago

DennisSchouwenaars commented 6 months ago

Describe the bug

When using a single custom bicep template file with multiple *.parameters.json files, having the "ParallelDeployMultipleTemplateParameterFiles" set to true. Trying to deploy a x amount of parameters files at once occasionally result in a "file in use" error message. We've seen this behavior having the throttle limit set to 5, 10 or 15. We use this option multiple times within our repo (and failures occur at different points depending on timing).

It seems the "Foreach-Object -Parallel -UseNewRunspace", locks the file at a certain time. And since the actual deployment file it uses for that given loop only exists once it gets locked at a certain point within the parallel loop (I assume the higher the throttle limit the more likely this behavior occurs). Suggested change would be to load the file in memory before the parallel loop or temp generate a temp copy of the file for the given job/run.

Steps to reproduce

  1. Having the following properties: "ParallelDeployMultipleTemplateParameterFiles", "AllowMultipleTemplateParameterFiles", "AllowMultipleTemplateParameterFiles" set to true, Throttle Limit set to (5, 10 or 15)
  2. Invoking a deployment causing multiple files pointing to the same template (number of parameters smaller or bigger than the number where the throttle limit is set to doesn't matter).
  3. Deployment might fail when a parallel job allocates the file at the same time.

Screenshots

image

Jefajers commented 6 months ago

Hi @DennisSchouwenaars, thanks for reporting this behavior. I will have a look at reproducing this and evaluate possible mitigations.

Jefajers commented 6 months ago

@DennisSchouwenaars, I have tried multiple combinations of Windows/Linux and I am not able to reproduce the issue as described.

However I can simulate conditions by creating a file lock during module execution and experience the different places that the module then fails to access the file.

I am currently thoroughly testing out logic that relieves the file lock in my simulation by storing the template content in-memory prior to deployment as you suggested above.