An exception occurred while analyzing template.

maikvandergaag commented 1 year ago

Describe the bug

We receive this error when we analyze bicep templates containing the metadata keyword. Removing the metadata keyword solves this problem, but we need this keyword for documentation and information purposes.

Information regarding this can be found in the reproduction steps.

We get the following errors:

Expected behavior

Template files are analyzed with the metadata section.

Reproduction Steps

Sample bicep

`metadata info = { name: 'RBAC Module' description: 'This module deploys a Role Based Access assignments' version: '1.0.0' author: 'Maik van der Gaag' }

param principalId string param roleDefinitionResourceId string

@description('This is the built-in Contributor role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#contributor') resource roleDefinition 'Microsoft.Authorization/roleDefinitions@2018-01-01-preview' existing = { scope: subscription() name: roleDefinitionResourceId }

resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = { name: guid(resourceGroup().id, principalId, roleDefinition.id) properties: { roleDefinitionId: roleDefinition.id principalId: principalId principalType: 'ServicePrincipal' } scope: resourceGroup() }`

When we analyze this with '.\TemplateAnalyzer.exe analyze-template D:\source\rbac.bicep -v'

Environment

Local and on Microsoft build agents
.Net 6

nonik0 commented 1 year ago

@maikvandergaag Thank you for reporting, we will look into this.

juju4 commented 1 year ago

I'm also getting this error when scanning the following Azure public repo and v0.5.1

https://github.com/Azure-Samples/azure-search-openai-demo

https://github.com/Azure/Azure-Sentinel/ (more the Detections but those in repo are in yaml, need the one converted to ARM)

    1 instance of: An exception occurred while analyzing template /home/vsts/work/1/s/DetectionsJSON/ASimAuthentication/Brute_force_attack_against_user_credentials__Uses_Authentication_Normalization_.json
    1 instance of: An exception occurred while analyzing template /home/vsts/work/1/s/DetectionsJSON/ASimAuthentication/User_login_from_different_countries_within_3_hours__Uses_Authentication_Normalization_.json
    1 instance of: An exception occurred while analyzing template /home/vsts/work/1/s/DetectionsJSON/ASimAuthentication/Potential_Password_Spray_Attack__Uses_Authentication_Normalization_.json

Any direction to help avoid those?

maikvandergaag commented 1 year ago

Any updates on this?

nonik0 commented 1 year ago

@maikvandergaag Very sorry for the delay, I was on parental leave! OK--so since you opened this issue we have had another minor release 0.5.X. It includes a bicep dependency update as well as other bicep scanning improvements.

I believe this fixes your issue as I am getting no error with your test templates with 0.5.X, but I do with 0.4.0. Please let me know if you are still seeing this issue after trying latest release.

nonik0 commented 1 year ago

@juju4 I took a look at the openai-demo repo's results and in that repo I see issues with invalid expressions causing parsing issues not am not getting any errors (i.e. preventing scanning of the file due to uncaught exception). Could you share more specifics of the templates that are giving you errors?

juju4 commented 1 year ago

It's more the exception/warning that are noisy and not actionable. may be an option to redirect those to a separate log that can be reported to project?

Full output on https://github.com/juju4/azure-search-openai-demo/actions/runs/5299769415/jobs/9592934762#step:11:694

    Running ARM Template Best Practice Analyzer 0.5.1
    ------------------------------------------------------------------------------
    /home/runner/work/_msdo/packages/nuget/Azure.Templates.Analyzer.CommandLine.linux-x64.0.5.1/tools/TemplateAnalyzer analyze-directory /home/runner/work/azure-search-openai-demo/azure-search-openai-demo --report-format sarif --output-file-path /home/runner/work/azure-search-openai-demo/azure-search-openai-demo/.gdn/.r/templateanalyzer/001/templateanalyzer.sarif
    Discovered 14 template-parameter pairs to analyze

    Directory: /home/runner/work/azure-search-openai-demo/azure-search-openai-demo
    Warning: An exception occurred while evaluating the properties of the resource named web
    Warning: An exception occurred while evaluating the properties of the resource named openai-role-backend
    Warning: An exception occurred while evaluating the properties of the resource named storage-role-backend
    Warning: An exception occurred while evaluating the properties of the resource named search-role-backend
    Warning: The parsing of the template output named AZURE_OPENAI_SERVICE failed
    Warning: The parsing of the template output named AZURE_FORMRECOGNIZER_SERVICE failed
    Warning: The parsing of the template output named AZURE_SEARCH_SERVICE failed
    Warning: The parsing of the template output named AZURE_STORAGE_ACCOUNT failed
    Warning: The parsing of the template output named BACKEND_URI failed
    Warning: The parsing of the template output named identityPrincipalId failed
    Warning: The parsing of the template output named uri failed
    Warning: The parsing of the template output named endpoint failed
    Warning: The parsing of the template output named endpoint failed
    Warning: The parsing of the template output named primaryEndpoints failed
    Warning: The parsing of the template output named endpoint failed
    Warning: The parsing of the template output named uri failed
    Warning: The parsing of the template output named primaryEndpoints failed
    Warning: An exception occurred while evaluating the properties of the resource named web
    Warning: An exception occurred while evaluating the properties of the resource named openai-role-backend
    Warning: An exception occurred while evaluating the properties of the resource named storage-role-backend
    Warning: An exception occurred while evaluating the properties of the resource named search-role-backend
    Warning: The parsing of the template output named AZURE_OPENAI_SERVICE failed
    Warning: The parsing of the template output named AZURE_FORMRECOGNIZER_SERVICE failed
    Warning: The parsing of the template output named AZURE_SEARCH_SERVICE failed
    Warning: The parsing of the template output named AZURE_STORAGE_ACCOUNT failed
    Warning: The parsing of the template output named BACKEND_URI failed
    Warning: The parsing of the template output named identityPrincipalId failed
    Warning: The parsing of the template output named uri failed
    Warning: The parsing of the template output named endpoint failed
    Warning: The parsing of the template output named endpoint failed
    Warning: The parsing of the template output named primaryEndpoints failed
    Warning: The parsing of the template output named endpoint failed
    Warning: The parsing of the template output named uri failed
    Warning: The parsing of the template output named primaryEndpoints failed

    Analyzed 14 files in the directory specified.

    Execution summary:
        The verbose mode (option -v or --verbose) can be used to obtain even more information about the execution.

        Summary of the warnings:
            2 instances of: An exception occurred while evaluating the properties of the resource named web
            2 instances of: An exception occurred while evaluating the properties of the resource named openai-role-backend
            2 instances of: An exception occurred while evaluating the properties of the resource named storage-role-backend
            2 instances of: An exception occurred while evaluating the properties of the resource named search-role-backend
            2 instances of: The parsing of the template output named AZURE_OPENAI_SERVICE failed
            2 instances of: The parsing of the template output named AZURE_FORMRECOGNIZER_SERVICE failed
            2 instances of: The parsing of the template output named AZURE_SEARCH_SERVICE failed
            2 instances of: The parsing of the template output named AZURE_STORAGE_ACCOUNT failed
            2 instances of: The parsing of the template output named BACKEND_URI failed
            2 instances of: The parsing of the template output named identityPrincipalId failed
            4 instances of: The parsing of the template output named uri failed
            6 instances of: The parsing of the template output named endpoint failed
            4 instances of: The parsing of the template output named primaryEndpoints failed

        34 Warnings
        0 Errors

    Wrote 31 results to /home/runner/work/azure-search-openai-demo/azure-search-openai-demo/.gdn/.r/templateanalyzer/001/templateanalyzer.sarif
    Tool run time: 28.7612185 seconds
    ------------------------------------------------------------------------------
    ARM Template Best Practice Analyzer completed with exit code 20
    ------------------------------------------------------------------------------

    Running ESLint 7.32.0.2
    ------------------------------------------------------------------------------
    /home/runner/work/_msdo/packages/node_modules/eslint/bin/eslint --no-eslintrc --config /home/runner/work/_msdo/packages/node_modules/eslint/lib/node_modules/@microsoft/eslint-plugin-sdl/config/required.js --ext .js --ext .ts --ignore-pattern *.d.ts --format /home/runner/work/_msdo/packages/node_modules/eslint/lib/node_modules/@microsoft/eslint-formatter-sarif/sarif.js --output-file /home/runner/work/azure-search-openai-demo/azure-search-openai-demo/.gdn/.r/eslint/001/eslint.sarif **/*.{js,ts}
    Tool run time: 0.7665677 seconds
    ------------------------------------------------------------------------------
    ESLint completed with exit code 1
    ------------------------------------------------------------------------------

  Process:
    Convert:
      Converting any raw tool logs to Sarif format ...
      Completed converting raw tool logs to Sarif format.
    Import:
      Importing results to the Guardian result repository...
      Saved file /home/runner/work/azure-search-openai-demo/azure-search-openai-demo/.gdn/r/.gdntoolsdata
      This file does not contain identifying data. It is safe to check into your repo. To hydrate this file with identifying data, run `guardian hydrate --help` and follow the guidance.
      Saved file /home/runner/work/azure-search-openai-demo/azure-search-openai-demo/.gdn/internal.gdnhistory
      Completed importing results to the Guardian result repository. 9 result file(s) were created in /home/runner/work/azure-search-openai-demo/azure-search-openai-demo/.gdn/r
  Break:
    Guardian is searching for results that meet the given criteria to break the build.
    Results Query Summary:
      Tool Filters (Include): bandit, terrascan, templateanalyzer, eslint
      Baselines: default
      Suppression Sets: default
      Policy: GitHub
Error:      1. TemplateAnalyzer Error TA-000017 - File: infra/main.json. Line: 549. Column 0. 
    Tool: TemplateAnalyzer: Rule: TA-000017 (AppServiceWebApp.UseLatestTLS). https://github.com/Azure/template-analyzer/blob/main/docs/built-in-rules.md/#ta-000017-latest-tls-version-should-be-used-in-your-web-app
    Web apps should require the latest TLS version.
Error:      2. TemplateAnalyzer Error TA-000017 - File: infra/main.json. Line: 558. Column 0. 
    Tool: TemplateAnalyzer: Rule: TA-000017 (AppServiceWebApp.UseLatestTLS). https://github.com/Azure/template-analyzer/blob/main/docs/built-in-rules.md/#ta-000017-latest-tls-version-should-be-used-in-your-web-app
    Web apps should require the latest TLS version.
Error:      3. TemplateAnalyzer Error TA-000017 - File: infra/main.json. Line: 592. Column 0. 
    Tool: TemplateAnalyzer: Rule: TA-000017 (AppServiceWebApp.UseLatestTLS). https://github.com/Azure/template-analyzer/blob/main/docs/built-in-rules.md/#ta-000017-latest-tls-version-should-be-used-in-your-web-app
    Web apps should require the latest TLS version.
Error:      4. TemplateAnalyzer Error AZR-000280 - File: infra/main.json. Line: 868. Column 0. 
    Tool: TemplateAnalyzer: Rule: AZR-000280 (Azure.Cognitive.PublicAccess). https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.Cognitive.PublicAccess/
    By default, public network access is enabled for a Cognitive Service account. Service Endpoints and Private Link can be leveraged to restrict access to PaaS endpoints. When access is restricted, access by malicious actor is from an unauthorized virtual network is mitigated.
Configure service endpoints and private links where appropriate.
Error:      5. TemplateAnalyzer Error AZR-000280 - File: infra/main.json. Line: 866. Column 0. 
    Tool: TemplateAnalyzer: Rule: AZR-000280 (Azure.Cognitive.PublicAccess). https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.Cognitive.PublicAccess/
    By default, public network access is enabled for a Cognitive Service account. Service Endpoints and Private Link can be leveraged to restrict access to PaaS endpoints. When access is restricted, access by malicious actor is from an unauthorized virtual network is mitigated.
Configure service endpoints and private links where appropriate.
Error:      6. TemplateAnalyzer Error AZR-000281 - File: infra/main.json. Line: 859. Column 0. 
    Tool: TemplateAnalyzer: Rule: AZR-000281 (Azure.Cognitive.ManagedIdentity). https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.Cognitive.ManagedIdentity/
    Cognitive Services must authenticate to Azure resources such storage accounts. To authenticate to Azure resources, Cognitive Services can use managed identities.
Using Azure managed identities have the following benefits:

Azure / template-analyzer