ARM Template Checker considers no controls for deployed resources as a failure

ankrause commented 5 years ago

Title

ARM Template Checker considers "no controls" as a failure

Description

There are a number of reasons why a template might not have anything that can be evaluated. In these scenarios, the ARM Template Checker task will write an error to output which fails the task unless the task is set to continue even on failure (resulting in partial success instead).

Since these tasks are intended to alert on or block potentially insecure or mis-configured deployments, this is unfortunate. Until one or more evaluatable policies and/or resources are included, the task either needs to be disabled or set to continue anyway which doesn't protect from future changes.

2019-08-08T06:21:01.8331196Z ================================================================================
2019-08-08T06:21:01.8331581Z AzSK Version: 3.15.0
2019-08-08T06:21:01.8332894Z ================================================================================
2019-08-08T06:21:01.8464474Z Method Name: Get-AzSKARMTemplateSecurityStatus
2019-08-08T06:21:01.8464633Z Input Parameters:
2019-08-08T06:21:01.8464745Z Key Value
2019-08-08T06:21:01.8464889Z --- -----
2019-08-08T06:21:01.8465033Z ARMTemplatePath <snip>\DeploymentTemplate.json
2019-08-08T06:21:01.8465196Z ParameterFilePath <snip>\Parameters\<params>.json
2019-08-08T06:21:01.8465310Z UseBaselineControls True
2019-08-08T06:21:01.8465431Z ================================================================================
2019-08-08T06:21:02.0033489Z ================================================================================
2019-08-08T06:21:02.0034551Z Starting analysis: [FileName: .\DeploymentTemplate.json]
2019-08-08T06:21:02.0035180Z --------------------------------------------------------------------------------
2019-08-08T06:21:02.0091041Z No controls have been evaluated for file: .\DeploymentTemplate.json
2019-08-08T06:21:02.0154741Z ================================================================================
2019-08-08T06:21:02.0177074Z Skipped file(s): 1
2019-08-08T06:21:02.0223989Z .\DeploymentTemplate.json
2019-08-08T06:21:02.0318365Z --------------------------------------------------------------------------------
2019-08-08T06:21:02.0687490Z One or more files were skipped during the scan.
2019-08-08T06:21:02.0740076Z Either the files are invalid as ARM templates or those resource types are currently not supported by this command.
2019-08-08T06:21:02.0742203Z Please verify the files and re-run the command.
2019-08-08T06:21:02.0742769Z For files that should not be included in the scan, you can use the '-ExcludeFiles' parameter.
2019-08-08T06:21:02.0743115Z --------------------------------------------------------------------------------
2019-08-08T06:21:02.0743391Z No controls have been evaluated for ARM Template(s).
2019-08-08T06:21:02.0743818Z --------------------------------------------------------------------------------
2019-08-08T06:21:02.0744106Z ** Next steps **
2019-08-08T06:21:02.0744388Z Look at the individual control evaluation status in the CSV file.
2019-08-08T06:21:02.0744686Z a) If the control has passed, no action is necessary.
2019-08-08T06:21:02.0744999Z b) If the control has failed, look at the control evaluation detail in the CSV file (LineNumber, ExpectedValue, CurrentValue, etc.) and fix the issue.
2019-08-08T06:21:02.0745347Z c) If the control status says 'Skipped', it means that you have chosen to skip certain controls using the '-SkipControlsFromFile' parameter.
2019-08-08T06:21:02.0745653Z For further details, refer: https://aka.ms/devopskit/cicd
2019-08-08T06:21:02.0745932Z --------------------------------------------------------------------------------
2019-08-08T06:21:02.0746595Z Status and detailed logs have been exported to: C:\Users\VssAdministrator\AppData\Local\Microsoft\AzSKLogs\ARMChecker\20190808_062101
2019-08-08T06:21:02.0746920Z ================================================================================
2019-08-08T06:21:05.6655092Z Cleaning logs from temp directory...
2019-08-08T06:21:05.8062570Z ##[error]No controls have been evaluated for ARM Template(s).
2019-08-08T06:21:05.9227313Z ##[section]Finishing: AzSK ARM Template Checker

As a note, the above logs are from a setup that works if I remove -UseBaselineControls. The template and its parameters are valid and controls can be evaluated if they are enabled.

Steps to reproduce

This can be repro'd locally with a simple deployment template:

Get-AzSKARMTemplateSecurityStatus -ARMTemplatePath "SampleDeployment.json"

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
  "contentVersion": "1.0.0.0",
  "parameters": {
  },
  "variables": {
    "ApplicationInsightsName": "MySample",
  },
  "resources": [
    {
      "type": "Microsoft.Insights/components",
      "apiVersion": "2014-08-01",
      "name": "[variables('ApplicationInsightsName')]",
      "location": "[resourceGroup().location]",
      "properties": {
        "ApplicationId": "[variables('ApplicationInsightsName')]",
        "Application_Type": "web",
        "Flow_Type": "Bluefield",
        "Request_Source": "rest"
      }
    }
  ]
}

Expected behavior

In my opinion, the error here should only be written if the task actually fails, policies can't be loaded from the server (e.g. organizational policies), the template/parameters specified are invalid, or any other true error scenario. If there are simply no resources to evaluate, no policies to evaluate, or no policies to evaluate for the specified resources, then the task should be considered a success.

It would also be beneficial if the error were more specific to the scenario encountered.

ankrause commented 5 years ago

This is also an issue for SVTs which say

2019-08-20T22:11:57.4161566Z ##[error]Security report not generated for provided resource details. Please check if task configurations are correct.
2019-08-20T22:11:57.4405736Z Cleaning logs from temp directory...
2019-08-20T22:11:57.5460430Z ##[error]Unable to perform security scan. Please check task configurations/variables.

My variables are correct, it's just that the target RG doesn't have a combination of resources vs controls that can be evaluated, resulting in 0 evaluations.

This error makes it difficult to enforce the usage of the task and prevent future non-compliance.

andrewholler commented 4 years ago

This issue affects our team as well and it is a little disappointing that this issue has got no feedback since being opened. We have the same issue in that we don't wish to disable the secure check or treat the errors as warnings in case of future additions to the security checks being ignored as they would show up as errors on a pipeline that always errors.

ganesh-msft commented 4 years ago

You can use and set variable FailTaskIfNoControlsScanned to 'false' for the task to not fail even if no controls were scanned. Please refer https://github.com/azsk/DevOpsKit-docs/blob/master/03-Security-In-CICD/Readme.md#advanced-cicd-scanning-capabilities-1 for more details.

andrewholler commented 4 years ago

@ganesh-msft Hi, the description of this control variable is:

"This variable is to control the behavior of the SVT extension in case of no controls scanned. For e.g., using this, one may choose to pass the task if it is configured to scan only 'High' severity control but there are no resources for which 'High' severity controls are applicable."

This issue is in regards to the ARM template security status functionality of AzSK and not SVTs. I also tested the AzSKARMTemplateChecker@4 task in a pipeline and set the pipeline variable, 'FailTaskIfNoControlsScanned':'false' and the task still failed on a template without any controls scanned.

RohitYadav-msft commented 4 years ago

Hi @aholler2 , Thanks for the details, we are able to repro it, we'll fix it in upcoming release

manvkaur commented 4 years ago

Hi Team, Facing this issue with multiple resource types. Any work arounds or expected release date of the fix?

andrewholler commented 3 years ago

Has this been addressed?

andrewholler commented 3 years ago

Has this issue been fixed in a release?

azsk / DevOpsKit