New-AzResourceGroupDeployment Fails to determine dynamic (template) parameters when using the TemplateUri

brwilkinson commented 3 years ago

Sample Template for demo purposed

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "foo": {
            "type": "string",
            "metadata": {
                "description": "description"
            }
        },
        "bar": {
            "type": "string",
            "metadata": {
                "description": "description"
            }
        }
    },
    "functions": [],
    "variables": {

    },
    "resources": [],
    "outputs": {
        "foo": {
            "type": "string",
            "value": "[parameters('foo')]"
        },
        "bar": {
            "type": "string",
            "value": "[parameters('bar')]"
        }
    }
}

Sample param file for Demo

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "bar": {
            "value": "AZE2"
        },
        "foo": {
            "value": "helo"
        }
    }
}

brwilkinson commented 3 years ago

Works as expected

New-AzResourceGroupDeployment @Params 

DeploymentName          : nestedTest
ResourceGroupName       : AZC1-BRW-ABC-RG-S1
ProvisioningState       : Succeeded
Timestamp               : 1/21/2021 11:44:56 PM
Mode                    : Incremental
TemplateLink            : 
                          Uri            : https://azc1brwhubg1saglobal.blob.core.windows.net/azc1-abc-stageartifacts-benwilk/templates-nested/nestedTest.json
                          ContentVersion : 1.0.0.0

Parameters              : 
                          Name             Type                       Value     
                          ===============  =========================  ==========
                          foo              String                     helo      
                          bar              String                     AZE2

Outputs                 :
                          Name             Type                       Value
                          ===============  =========================  ==========
                          foo              String                     helo
                          bar              String                     AZE2

DeploymentDebugLogLevel :

brwilkinson commented 3 years ago

Works as expected

New-AzResourceGroupDeployment @Params 

ProvisioningState       : Succeeded
Timestamp               : 1/21/2021 11:45:19 PM
Mode                    : Incremental
TemplateLink            :
                          Uri            : https://azc1brwhubg1saglobal.blob.core.windows.net/azc1-abc-stageartifacts-benwilk/templates-nested/nestedTest.json
                          ContentVersion : 1.0.0.0

Parameters              : 
                          Name             Type                       Value
                          ===============  =========================  ==========
                          foo              String                     abc
                          bar              String                     123

Outputs                 :
                          Name             Type                       Value
                          ===============  =========================  ==========
                          foo              String                     abc
                          bar              String                     123

DeploymentDebugLogLevel :

brwilkinson commented 3 years ago

Works as expected###

New-AzResourceGroupDeployment @Params 

DeploymentName          : nestedTest
ResourceGroupName       : AZC1-BRW-ABC-RG-S1
ProvisioningState       : Succeeded
Timestamp               : 1/21/2021 11:46:06 PM
Mode                    : Incremental
TemplateLink            :
Parameters              :
                          Name             Type                       Value
                          ===============  =========================  ==========
                          foo              String                     helo
                          bar              String                     AZE2

Outputs                 :
                          Name             Type                       Value
                          ===============  =========================  ==========
                          foo              String                     helo
                          bar              String                     AZE2

DeploymentDebugLogLevel :

brwilkinson commented 3 years ago

Works as expected

Now update the parameter file to remove 1 parameter to pass in 'foo' dynamically at runtime.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "bar": {
            "value": "AZE2"
        }
    }
}

New-AzResourceGroupDeployment @Params 

DeploymentName          : nestedTest
ResourceGroupName       : AZC1-BRW-ABC-RG-S1
ProvisioningState       : Succeeded
Timestamp               : 1/21/2021 11:47:10 PM
Mode                    : Incremental
TemplateLink            :
Parameters              :
                          Name             Type                       Value
                          ===============  =========================  ==========
                          foo              String                     abc
                          bar              String                     AZE2

Outputs                 :
                          Name             Type                       Value
                          ===============  =========================  ==========
                          foo              String                     abc
                          bar              String                     AZE2

brwilkinson commented 3 years ago

This now breaks . . . using TemplateURI

New-AzResourceGroupDeployment @Params

New-AzResourceGroupDeployment:
Line |
  10 |  New-AzResourceGroupDeployment @Params
     |                                ~~~~~~~
     | A parameter cannot be found that matches parameter name 'foo'.

Also when update when you remove foo . . .

New-AzResourceGroupDeployment @Params 

New-AzResourceGroupDeployment: 
Line |
  10 |  New-AzResourceGroupDeployment @Params
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | 3:50:59 PM - Error: Code=InvalidTemplate; Message=Deployment template validation failed: 'The value for the template parameter 'foo' at line '5' and column '16' is not provided. Please see https://aka.ms/resource-manager-parameter-files for usage details.'.

New-AzResourceGroupDeployment:
Line |
  10 |  New-AzResourceGroupDeployment @Params
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The deployment validation failed

brwilkinson commented 3 years ago

I demoed with splatting for simplicity, however, same goes for using the parameters inline.

New-AzResourceGroupDeployment -queryString $queryString -TemplateUri $TemplateURi -ResourceGroupName $rgName -TemplateParameterFile $myParams -foo abc

brwilkinson commented 3 years ago

@alex-frankel just wanted to tag you on this one, since I didn't see anyone pick up this issue.

This is a blocker for me deploying via template specs, I wanted to keep moving forward on my testing/implementation.

alex-frankel commented 3 years ago

@Xynoclafe is working on a fix. We will update this with the azure-powershell PR once it's out.

Xynoclafe commented 3 years ago

@brwilkinson @alex-frankel The PR was merged. The fix should be included with the next release. https://github.com/Azure/azure-powershell/pull/14132

brwilkinson commented 3 years ago

Thank you @alex-frankel @Xynoclafe ✅

brwilkinson commented 3 years ago

Hi @Xynoclafe

just checking on the new modules

Did the fix not make it into 3.3.0 [Az.Resources]?

I still see the same issue..

Xynoclafe commented 3 years ago

Hi @brwilkinson The fix should be present in Az Modules 5.6.0. In the screenshots you added, my guess is that the error is due to the "?" at the start of the QueryString. -QueryString takes the SASToken without the preceding "?".

brwilkinson commented 3 years ago

oops deleted my comment, take 2.

Thanks @Xynoclafe good catch.

Yes it does work as expected without the ?

I did test this before posting yesterday and it works with the ? when not using dynamic params.

Not sure why this doesn't fail ? with the ?

It would be nice if the string was normalized to remove the ?, since when you copy it from storage explorer etc, it is always included.

obviously the same with dynamic, which is what I showed yesterday fails....

Anyway, thank you, I look forward to testing this more on Template Specs.

alex-frankel commented 3 years ago

Closing since the initial issue is resolved. @brwilkinson we are planning to normalize this behavior I believe. @snarkywolverine - did we end up implementing that already?

snarkywolverine commented 3 years ago

Yes, you can now either include or omit the leading ? for the queryString.

brwilkinson commented 3 years ago

Thanks @snarkywolverine

we may have found a scenario that is not stripping the ?

When using a dynamic param... it doesn't normalize.

There was a recent update to make this work, so perhaps a regression on that functionality?

This was the update Azure/azure-powershell#14132

Xynoclafe commented 3 years ago

@brwilkinson The behavior right now is as expected since dynamic parameters are resolved before the "?" is normalized. We can definitely add the "?" normalization to the dynamic parameters scenario as well in a future version.

SvenAelterman commented 3 years ago

@Xynoclafe I have Az.Resources 3.4.1 in an Azure Automation account. When using the New-AzResourceGroupDeployment cmdlet in a PowerShell runbook, I get an error that the parameter isn't specified when it's not in the parameters file. When I specify the parameter in the parameters file and then specify it as a dynamic parameter, the dynamic parameter value doesn't override the value from the template file (but no error is thrown).

Here's what my command looks like:

New-AzResourceGroupDeployment -ResourceGroupName "rg-filesynctoblob-demo-eastus-01" `
    -Name "filesynctoblob-aci" `
    -fileShareSasUri $ShareSASURI -blobSasUri $ContainerSASURI `
    -commandParameter $command `
    -QueryString $aciTemplateSASURI `
    -TemplateUri $templateUri -TemplateParameterUri $parameterUri

The $templateUri does not contain the SAS token querystring; therefore I specify the QueryString param. However, the QueryString param does not seem to affect the -TempalteParamaterUri param, so my SAS token is in $parameterUri.

Azure / template-specs