Bicep "concat()" doesn't play nicely with "what-if"

[rickbatka]

Describe the noise

Resource type (i.e. Microsoft.Storage/storageAccounts) "Microsoft.App/containerApps" in my case, but also any others if you use concat() in Bicep to create a collection

apiVersion (i.e. 2019-04-01) All

Client (PowerShell, Azure CLI, or API) az cli, using az deployment group what-if with a bicep input file

Relevant ARM Template code (we only need the resource object for the above resourceType and apiVersion, but if it's easier you can include the entire template

(This is Bicep, not ARM)...

secrets: concat(myContainerApp.additionalSecrets, [
          {
            name: 'another-secret'
            value: someValue
          }
...
]

Expected response (i.e. "I expected no noise since the template has not been modified since the resources were deployed)

I expect no change because the result of the concat() operation hasn't changed. I don't change either of the source arrays (they both come from other places in the Bicep file, and they just serve as a common base collection for lots of apps that share common config).

Current (noisy) response (either include a screenshot of the what-if output, or copy/paste the text)

No matter how many times I deploy the exact same Bicep file, to the same Azure target, with the exact same inputs, it always reports ALL values changing if the concat() function was used. Also, the "before" and "after" results are formatted completely differently:

~ properties.template.containers: [
      ~ 0:

        ~ env: [
            0:

              name:  "redacted"
              value: "redacted"

            1:

              name:  "redacted"
              value: "redacted"

            2:

              name:  "redacted"
              value: "redacted"

            3:

              name:  "redacted"
              value: "redacted"

            4:

              name:  "redacted"
              value: "redacted"

            5:

              name:  "redacted"
              value: "redacted"

            6:

              name:  "redacted"
              value: "redacted"

            7:

              name:  "redacted"
              value: "redacted"

            8:
              name:  "redacted"
              value: "redacted"

            9:

              name:  "redacted"
              value: "redacted"

            10:

              name:  "redacted"
              value: "redacted"

            11:

              name:  "redacted"
              value: "redacted"

            12:

              name:  "redacted"
              value: "redacted"

            13:

              name:  "redacted"
              value: "redacted"

            14:

              name:  "redacted"
              value: "redacted"

            15:

              name:  "redacted"
              value: "redacted"

            16:

              name:  "redacted"
              value: "redacted"

            17:

              name:  "redacted"
              value: "redacted"

            18:

              name:  "redacted"
              value: "redacted"

            19:

              name:  "redacted"
              value: "redacted"

            20:

              name:  "redacted"
              value: "redacted"

            21:

              name:  "redacted"
              value: "redacted"

          ] => "[concat(variables('tenantContainerApps')[copyIndex()].additionalEnvVars, createArray(createObject('name', 'redacted', 'value', if(equals(parameters('environmentName'), 'staging'), 'redacted.azurecontainerapps.io', 'redacted')), createObject('name', 'redacted', 'secretRef', 'redacted'), createObject('name', 'redacted', 'secretRef', 'redacted'), createObject('name', 'redacted', 'value', 'false'), createObject('name', 'redacted', 'value', 'redacted'), createObject('name', 'redacted', 'secretRef', 'redacted'), createObject('name', 'redacted', 'secretRef', 'redacted'), createObject('name', 'redacted', 'secretRef', 'redacted'), createObject('name', 'redacted', 'value', reference(resourceId('Microsoft.App/containerApps', 'redacted'), '2022-03-01').configuration.ingress.fqdn), createObject('name', 'redacted', 'value', 'redacted'), createObject('name', 'redacted', 'value', 'redacted'), createObject('name', 'redacted', 'value', parameters('redacted')), createObject('name', 'redacted', 'value', parameters('redacted')), createObject('name', 'redacted', 'secretRef', 'redacted'), createObject('name', 'redacted', 'secretRef', 'redacted'), createObject('name', 'redacted', 'value', parameters('redacted'))))]"

Additional context

It also mangles the output if you use conditionals - you can see the if() logic baked into the what-if output pasted above.

I also reported this to the AZ CLI team (ticket here) a month ago, before I knew about this repo specific to what-if.

I built a whole infrastructure around Bicep and I'm really regretting it. I either can't use Bicep, or I can't use what-if. So far, I've been using Bicep and ignoring the output of what-if. Because I use Bicep templating extensively, the what-if output is essentially all noise for me - it reports changes to every resource.

I feel like I'm being punished for using the Microsoft promoted tooling here. I regret adopting Container Apps and Bicep so far. They could be great, if some of these inconsistencies could be worked out.

[rickbatka]

I really hope someone from the team can look into this. The what-if tool doesn't seem to really work with Bicep templating.

[rickbatka]

Any chance this will be addressed? I get very nervous every time I deploy any change to my infrastructure because of this. The what-if tool always tells me that everything is changing.

Is there any known workaround? Can I compile my bicep files into ARM json and then get better results? Or should I just stop using Bicep?

[alex-frankel]

Hi Rick - what-if is a best effort tool and there are a variety of cases where you can get noise like this. Can you confirm that removing concat() gets rid of the noise?

Bicep files are already transpiled to ARM Template JSON prior to running what-if, so moving back to ARM templates won't change the quality of the what-if response.

We are starting work on a What-If vNext for which we will share some details in the next 1-2 months. The hope is that we can make some archiectural changes that make what-if more reliable across the board. Tagging @sydkar as FYI.

Of course, switching to a tool like Terraform is also an option where the plan command is more reliable, but that is a significant migration and comes with other tradeoffs.

[rickbatka]

Thanks, @alex-frankel , I'm glad to get a response and I appreciate the candor about the "vNext" project.

Consider this my vote for better support of dynamic ARM and Bicep templates for the next version!

Without it, we struggle to see how we can use Bicep in a sane way. Either we pre-process our templates ourselves (removing the value of Bicep as a templating language), or we abandon "what-if" entirely (which is what we've done so far). This means when we deploy any change to our production infrastructure, we must deploy the whole thing, and cross our fingers that nothing unexpected is changing.

It's harrowing to say the least, especially as we build more and more in the Azure ecosystem.

As a bit of extra context - we use Azure Container Apps (your current most recommended solution for medium-size containerized apps). But, Azure Container Apps can't be deployed through the Azure Portal (UI) if they use volume mounts. So we are forced to use ARM, Bicep, or Bash scripts that use the Azure CLI to deploy our apps. We chose Bicep / ARM, and it has not been very smooth so far.

If we move to CLI, we will lose the benefits of IaC.

Moving to terraform is a tough pill to swallow, when it seems like Bicep is the best tool for the job and is 80% of the way there.

Just some food for thought, I know you have your own priorities. But this would make a world of difference to this one small startup.

[rickbatka]

I forgot to answer your other question head-on: yes, if we remove all uses of concat() and other dynamic methods, the what-if results are more correct. It's just that with our setup (lots of similar-but-slightly different container apps), the repeated code is enough to make this a non-starter.

If we could add a step to our process that compiled to ARM with the collections unrolled, that would be an acceptable workaround. It sounds like that isn't possible at the moment though - please correct me if I'm wrong.

[alex-frankel]

Thanks for the extra detail. There is no way to evaluate all of the expressions prior to a deployment if the expression includes any runtime properties (like myContainerApp.additionalSecrets appears to be).

FWIW, even if what-if was perfect, I wouldn't recommend relying on it as a perfect prediction of what is going to happen. We recommend instrumenting an integration test of some kind for IaC deployments such that you can do a full deployment and evaluate the results of that running deployment before verifying it is ready for production. That's not to say that having an accurate what-if would not be valuable in and of itself. I am hoping we make some material progress over the next six months.