Error in extracting test results from inner templates

[mikanygTC]

I have an issue in a rather advanced bicep template, consisting of a main template with a dozen child templates. I am using the latest version of the ARM-TTK xplat extension (v1.2.0) and using the az bicep option (with bicep extension being up-to-date).

The error I am seeing is:

##[error]Write-Error: C:\agents\agent01\_work\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.0\arm-ttk\Test-AzTemplate.ps1:785
##[error]Write-Error: C:\agents\agent01\_work\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.0\arm-ttk\Test-AzTemplate.ps1:785
##[error]Line |
##[error]Line |
##[error] 785 |                  Test-FileList # we just call it directly.
##[error] 785 |                  Test-FileList # we just call it directly.
##[error]     |                  ~~~~~~~~~~~~~
##[error]     |                  ~~~~~~~~~~~~~
##[error]     | Could not extract inner templates for 'C:\agents\agent01\_work\6\s\templates\bicep\maintemplate.json'.
##[error]     | Could not extract inner templates for 'C:\agents\agent01\_work\6\s\templates\bicep\maintemplate.json'.
##[error]
##[error]

I have set the following flags for the task:

     - task: RunARMTTKTestsXPlat@1
        displayName: Run ARM TTK Tests
        inputs:
          templatelocation: '$(Build.SourcesDirectory)/templates/bicep'
          resultLocation: '$(Build.SourcesDirectory)/ttk-results'
          skipTests: 'Parameter Types Should Be Consistent,URIs Should Be Properly Constructed'
          mainTemplates: 'maintemplate.bicep'
          allTemplatesMain: false
          cliOutputResults: false
          ignoreExitCode: true
          usePSCore: true
          useAzBicep: true

But the task still fails even though I set ignoreExitCode: false and the task also outputs the same no matter what I set cliOutputResults to.

Any thoughts on what could be done to not have it fail. The bicep templates are compiled just fine, the reason for supressing tests is that bicep generated ARM template contains some constructs that ARM TTK does not allow, so we just ignore those.

[sam-cogan]

Have you tested running the TTK directly against the files to see if that fails as well? This will help determine if it is a problem with the extension or the TTK.

Is it possible for you to provide a copy of the templates so I can replicate the problem, sanitised if needed? If you'd prefer, feel to email them to mail at samcogan dotcom.

The ignorExitCode setting is working as expected here, as that is only meant to supress failed tests, rather than any type of error.

[mikanygTC]

Just tested the ARM template (generated by bicep) against ARM TTK 0.22 directly, and it works without failures, just a few failed test cases, all due to bicep generated ARM code not fully complying with ARM TTK. Some of the failed test cases might even be "hard to fix bugs" in ARM TTK. E.g. I have a an output parameter from one template being referenced as input parameter in another, and the child template input parameter is generated by bicep like this:

"keyVaultUri": {
            "value": "[reference(extensionResourceId(format('/subscriptions/{0}/resourceGroups/{1}', subscription().subscriptionId, parameters('resourceGroupBase')), 'Microsoft.Resources/deployments', 'base_resources'), '2020-10-01').outputs.keyVaultUri.value]"
          },

which causes the test case "URIs Should Be Properly Constructed" to fail, due to the parameter name having Uri/Url in it, and the value contains format(), event though in this case its perfectly valid.

So the failure I am seeing in this issue seems to be related to parsing the test results in the extension. I cannot share the template here directly. Will see if I can repro it locally. Any tips on how to easily run the extension locally against the bicep template?

[mikanygTC]

My first attempts at running again ARM TTK showed no failures, but I was apparently calling it different from the extension, I see the same issue coming from ARM TTK as mentioned in this issue when using same parameters for Test-AzTemplate.ps1. I'll dig a bit more.

[mikanygTC]

Ok I have some more information.

There is inconsistency between parameters in ARM TTK Test-AzTemplate.ps1 and how its being called in Test-FolderContents function in invoke-ttk.psm1.

The parameter -mainTemplate $filter is actually called MainTemplateFile in Test-AzTemplate.ps1.

Another thing I tested out was that by executing ARM TTK using these parameters it works as expected.

Test-AzTemplate -TemplatePath D:\templates\bicep -File maintemplate.json

with output as: Pass : 28 Total : 33 Fail : 5

But calling it like in Test-FolderContents function in invoke-ttk.psm1 using these parameter it gives the error Could not extract inner templates for 'D:\templates\bicep\maintemplate.json'.

Test-AzTemplate -TemplatePath D:\templates\bicep\maintemplate.json

with output as: Pass : 492 Total : 532 Fail : 40

The lines here

if ($mainTemplate) {
        $results = Test-AzTemplate -TemplatePath "$folder\$filter" -Skip $Skip -Test $Test -mainTemplate $filter -ErrorAction  Continue
    }
    else {
        $results = Test-AzTemplate -TemplatePath "$folder\$filter" -Skip $Skip -Test $Test -ErrorAction Continue
    }

should be something like below, but needs verification.

if ($mainTemplate) {
        $results = Test-AzTemplate -TemplatePath $folder -File $filter -Skip $Skip -Test $Test -MainTemplateFile $filter -ErrorAction  Continue
    }
    else {
        $results = Test-AzTemplate -TemplatePath $folder -Skip $Skip -Test $Test -ErrorAction Continue
    }

[sam-cogan]

Thanks, that's really useful info. Let me dig into that and get back to you.

[sam-cogan]

@mikanygTC any chance you can provide an example bicep file that causes this error?

[mikanygTC]

@sam-cogan I will try to get time to provide a smaller generic template that can reproduce the error.

[mikanygTC]

@sam-cogan finally got around to putting together a small template to reproduce the error. I have attached a main template and a child template (module). There is a test failure in the child template and this causes the errors described earlier and the task always ends up in a failed state unless continueOnError: true.

I use the following Azure Pipeline task to get to the failure:

- task: RunARMTTKTestsXPlat@1
  displayName: Run ARM TTK Tests
  continueOnError: true
  inputs:
    templatelocation: '$(Build.SourcesDirectory)/templates/bicep'
    resultLocation: '$(Build.SourcesDirectory)/ttk-results'
    skipTests: 'URIs Should Be Properly Constructed,Parameter Types Should Be Consistent,Outputs Must Not Contain Secrets'
    mainTemplates: 'maintemplate.bicep'
    allTemplatesMain: false
    cliOutputResults: false
    ignoreExitCode: true
    usePSCore: true
    useAzBicep: true

Its run on a self-hosted agent win 11 and have installed azure CLI 2.45 and bicep 0.14.85. It was recently upgraded but failed the same in previous versions of CLI 2.44.x and bicep 0.13.x

repro-bicep-templates.zip

[mikanygTC]

Just saw the same error in a different repo after refactoring a few seperate templates into the following structure:

maintemplate.bicep
modules/childtemplate1.bicep
modules/childtemplate2.bicep
modules/childtemplate3.bicep

maintemplate links child templates using module reference in subfolder.

I suspect the error is related to this, because otherwise the content is totally unrelated to the uploaded repro

[mikanygTC]

@mikanygTC any chance you can provide an example bicep file that causes this error?

@sam-cogan any plans for looking into this issue? I provided a repro here: https://github.com/sam-cogan/arm-ttk-extension-xplatform/issues/16#issuecomment-1440939256