[BUG] [unexpected behavior] Failing to validate IaC using centralized custom PSRule rules and Baselines

[Marc013]

Existing rule

No response

Description of the issue

tl;dr; Based on my tests (see below) it proves it is not possible to have a central location for custom PSRule rules and baselines. Each repo containing Bicep or ARM templates require the custom PSRule rules and ps-rule.yaml configuration file to be present within that repo. This is by far a desired solution as it means PSRule code is going to be copied to each repo making it impossible to maintain it. I can’t escape the feeling I’m doing something wrong. But what??

Can you please help!?

Error messages

See test results references in Reproduction section.

Reproduction

My setup: Repo ‘IaC-validation’ contains the PSRule solution with custom PSRule rules.

• https://github.com/Marc013/IaC-validation Repo ‘myBicepSolution’ contains my Bicep solution which is to be validated.
• https://github.com/Marc013/myBicepSolution

Requirements: PSRule rules may not to be copied to the Bicep solutions repo. The Resource Manager parameter file is used to specify the arguments for mandatory parameters. The Resource Manager parameter file contains a reference to the corresponding Bicep file via metadata. Path to the corresponding Bicep file is relative (“metadata”: { "template": "./storageAccount.bicep" } to support validation in Azure pipeline. It is not desirable to have a PSRule configuration file in the Bicep solutions repo.

Goal: Validate IaC in repo ‘myBicepSolution’ using all custom PSRule rules including build in ‘Azure.Template.’ and ‘Azure.Deployment.’ rules located in repo ‘IaC-validation’. These custom rules consist of rules created using Azure policy assignment initiatives (JSON) and rules created in PowerShell.

My implementation: PSRule options (and configuration) are defined using cmdlet New-PSRuleOption. A PSRule baseline (myFirstBaseline) is created referencing every build in ‘Azure.Template.’ and ‘Azure.Deployment.’ and to include local rules. IaC validation (Assert-PSRule) is to be executed within the repo ‘myBicepSolution’ directory so the relative path in the parameters can successfully be resolved.

Tests: For all tests I used PowerShell script C:\Temp\IaC-validation\src\scripts\Run-PSRuleTest.ps1 The tests build upon the previous one. All changes made in previous tests are present in subsequent tests.

1. Using baseline ‘My.Guardrails1’ (.ps-rule\My.Guardrails.Baseline.Rule.yaml) the custom rules, ‘Azure.Template.’, ‘Azure.Deployment.’ AND PSRule for Azure build-in rules are validated. The PSRule for Azure build-in rules should not have been validated! Results available in ‘.\IaC-validation\testResults\Result-My.Guardrails1.txt'
2. Using baseline ‘My.Guardrails2’ (.ps-rule\My.Guardrails.Basline.Rule.jsonc) produces the exact same result.
3. Using baseline ‘TestBaseline1’ (.ps-rule\TestBaseline1.Baseline.Rule.yaml) also produces the exact same result.
4. After copying the PSRule baseline files to repo ‘myBicepSolution’ ..ps-rule (which is not desirable) the outcome is still the same. Get-PSRuleBaseline can find all 3 baselines.
5. After copying the custom rules and PSRule baseline files to repo ‘myBicepSolution’ ..ps-rule (which is not desirable) and modifying command Assert-PSRule to use the local rules the outcome is again the same.
6. Added PSRule configuration ps-rule.yaml in repo ‘myBicepSolution’ (this is not desirable), still the same outcome.
7. Using parameter -Baseline instead of defining it in the options of the PSRuleOptions. Baseline ‘TestBaseline1’ is used. However, the local custom policies are not used despite the baseline defining ‘includeLocal: true’ and having all the custom rules explicitly specified. Results available in ‘.\IaC-validation\testResults\Result-My.TestBaseline1.txt’,
8. Using ‘My.Guardrails2’ (no changes to the JSON file made). This time the baseline is used however, none of the explicitly specified custom rules are validated. Results available in ‘.\IaC-validation\testResults\Result-My.Guardrails2.txt’,
9. Using My.Guardrails1’ (no changes to the JSON file made). The exact same result as the previous test.
10. Explicitly defined all custom rules in ps-rule.yaml to be included. Not specifying any baseline. Custom rules and all build in rules are validated. The Build in rules (apart from the template and deployment rules) should not have been validated as that is not desired. Results available in ‘.\IaC-validation\testResults\Result-Using-ps-rule.yaml-with-all-custom-rules.txt’
11. Explicitly excluded every build in (PSRule.Rules.Azure) rule in ps-rule.yaml. Exact same result as in test 10.
12. Running Assert-PSRule without parameter -Option. All configuration specified in ps-rule.yaml. Nothing validated as no matching rules are found.
13. Running command Assert-PSRule -InputPath C:\Temp\myBicepSolution\storageAccount\storageAccount.parameters.json -Module PSRule.Rules.Azure -Format File. Desired outcome obtained! The command was executed in Windows terminal.
14. Running command of test 13 via script. Same outcome as in test 13. Which is positive.
15. Using parameter -Baseline My.Guardrails3. All excluded and included rules are specified in this baseline. Section Rule is commented out in ps-rule.yaml. Running command Assert-PSRule -InputPath C:\Temp\myBicepSolution\storageAccount\storageAccount.parameters.json -Module PSRule.Rules.Azure -Format File -Baseline My.Guardrails3. Only Azure.Template and Azure. Deployment rules are evaluated. All other rules are skipped.

Conclusion: PSRule requires custom rules to be withing the same repo as the Bicep/ARM template that is to be validated. In the same repo PSRule configuration file ps-rule.yaml is required. I won’t dive into things that don’t seem to work (or at least not for me) as it is already in detail described above.

Version of PSRule

2.9.0

Version of PSRule for Azure

1.30.3

Additional context

Provided in section Reproduction in My setup:.

[BernieWhite]

Hi @Marc013. Have a look at: https://microsoft.github.io/PSRule/v2/authoring/packaging-rules/

1. You want to add the rules from https://github.com/Marc013/IaC-validation into a module by changing the structure a bit instead of using the default .ps-rule/ directory add a directory for the module such as MyGuardrails.
2. Add a manifest.
3. In the manifest you will also want to take a dependency on PSRule.Rules.Azure by setting the RequiredModules property.

When executing tests from https://github.com/Marc013/myBicepSolution, you need to install/ import the module. If you are publishing the module you can Install-Module to install it prior or if you are not publishing the module anywhere you can git clone or use a sub-module to copy the module down then Import-Module to load the module prior to calling PSRule.

[Marc013]

@BernieWhite, Thank you very much for your quick reply and solution. I'll most definitely dive into the creation of a pwsh module for this.

[Marc013]

@BernieWhite

Attempting to create a module (and failing miserably 😭)

Can you please help me getting Assert-PSRule to validate my custom rules and PSRule.Rules.Azure rules ‘Azure.Template.*’ and ‘Azure.Deployment.*’?

In both repositories I create a new branch for the creation of the module and testing of it.

Repo	Branch
IaC-validation	MyGuardrails-pwsh-module
myBicepSolution	Use-pws-module-MyGuardrails

I created a pwsh module for the custom rules (which are in JSON format and 1 in pwsh) as you suggested.
In this module I adhered to your documentation as much as possible. So, I renamed ‘My.Guardrails.Baseline.Rule.yaml’ to ‘Baseline.Rule.yaml’.
Baseline TestBaseline1 and My.Guardrails3 is added to ‘Baseline.Rule.yaml’ (now containing 3 baselines) reducing the number of files in the module.

First test attempts

For each baseline I ran a test Results:

1. testResultsUsingModule\ModelTestResults-My.Guardrails1.txt
2. testResultsUsingModule\ModelTestResults-TestBaseline1.txt
3. testResultsUsingModule\ModelTestResults-My.Guardrails2.txt
4. testResultsUsingModule\ModelTestResults-My.Guardrails3.txt

Before each test I adjusted the baseline name in ‘modules\MyGuardrails\rules\Config.Rule.yaml’. Every test is run in its own pwsh terminal. Executed command for testing:

Assert-PSRule -InputPath .\storageAccount\storageAccount.parameters.json -Module MyGuardrails -Format File

None of the tests validated any of the rules 😢

Second test attempt

I copied module MyGuardrails to my default PowerShell module directory ('C:\Users\\\\\Documents\PowerShell\Modules\MyGuardrails').

Using baseline My.Guardrails3 I ran a new test in a new pwsh terminal.
The command used for this test is the same as specified above.

Unfortunately, the result was exactly the previous test results.

[WARN] Could not find a matching rule. Please check that Path, Name and Tag parameters are correct.

Third test attempt

Nothing has changed. Module MyGuardrails is still available in my default PowerShell module directory with baseline My.Guardrails3 defined.

This time I specified parameter -Options.

Command with configuration settings

$psRuleConfigurationSettings = @{
    AZURE_BICEP_FILE_EXPANSION         = $true
    AZURE_BICEP_FILE_EXPANSION_TIMEOUT = 60
    AZURE_BICEP_PARAMS_FILE_EXPANSION  = $true
    AZURE_PARAMETER_FILE_EXPANSION     = $true
    AZURE_PARAMETER_FILE_METADATA_LINK = $true
}

$psRuleOptions = New-PSRuleOption -Configuration $psRuleConfigurationSettings

Assert-PSRule -InputPath .\storageAccount\storageAccount.parameters.json -Module MyGuardrails -Format File -Option $psRuleOptions

Again the same results.

[WARN] Could not find a matching rule. Please check that Path, Name and Tag parameters are correct.

Fourth test attempt

For this test I copied module PSRule.Rules.Azure to MyGuardrails2 and:

1. Renamed version directory to '0.1.0'
2. Removed al files from version directory except 'PSRule.Rules.Azure.psd1' which I renamed to 'MyGuardrails2.psd1'
3. Adjusted the module manifest by providing a new guid and having it reflect module name 'MyGuardrails2'
4. Removed all rules
5. Added my custom rules (.JSONC and .ps1)
6. Removed directories '_manifest', 'en-AU', 'en-GB', 'en-US'
7. Emptied directory 'en' and placed markdown of my custom pwsh # Name of rule
8. Emptied directory 'rules' and placed my custom rule files (.jsonc & .ps1) and config & baseline yaml files
9. Copied module 'MyGuardrails2' to the PowerShell default module directory.
10. Executed command:

Assert-PSRule -InputPath .\storageAccount\storageAccount.parameters.json -Module MyGuardrails2 -Format File

Results remain the same.

[WARN] Could not find a matching rule. Please check that Path, Name and Tag parameters are correct.

Fifth attempt as I remembered pwsh module PowerShell-yaml

1. Copy version directory 0.1.0 to 0.2.0

2. Converted JSON rules to YAML using function ConvertTo-Yaml

   $customRules = Get-Content -Path C:\git\Private\_PSRuleTesting\IaC-validation\modules\MyGuardrails\0.2.0\rules\definitions-Guardrails-000.Rule.jsonc -Raw | ConvertFrom-Json -Depth 100
   
   $customRules | ConvertTo-Yaml -OutFile C:\git\Private\_PSRuleTesting\IaC-validation\modules\MyGuardrails2\0.2.0\rules\definitions-Guardrails-000.Rule.yaml

3. Removed 'definitions-Guardrails-000.Rule.jsonc'

4. Updated module version in manifest to '0.2.0'

5. Updated the rules in yaml file by adding a rule separator and synopsis (with pre1fix '[YAML]' in the description) and adjusting the indent

6. Copied version 0.2.0 to the PowerShell default module directory and executed command

7. Deleted directory version 0.1.0

8. Running test

Get-PSRule -Module MyGuardrails2 -ListAvailable

RuleName                            ModuleName  Synopsis
--------                            ----------  --------
My.Resource.MandatoryTags                       Resources not having the mandatory tags should be rejected.
Guardrails-000.Policy.231f0045f18f              [YAML]Storage accounts should have infrastructure encryption
Guardrails-000.Policy.9f15257d13f6              [YAML]Azure Stream Analytics jobs should use customer-managed keys to encrypt data
. . .

Assert-PSRule -InputPath .\storageAccount\storageAccount.parameters.json -Module MyGuardrails2 -Format File

Results remain the same.

[WARN] Could not find a matching rule. Please check that Path, Name and Tag parameters are correct.

[Marc013]

@BernieWhite, Could you perhaps shed some light on this?

[BernieWhite]

@Marc013 Working back with your initial module in modules/MyGuardrails.

Update My.Guardrails3 after includeLocal: true add tag: { }.

Add configuration to ps-rule.yaml in the myBicepSolution repo.

configuration:
  AZURE_PARAMETER_FILE_EXPANSION : true
  AZURE_PARAMETER_FILE_METADATA_LINK : true

Update your command line to include the -Baseline parameter.

assert-psrule -m MyGuardrails -f . -Format File -Baseline My.Guardrails3

---

There is a few PSRule bugs here. Ideally you should not need these additional steps. We're making some improvements in PSRule v3 that should fix these issues in cross-module scopes when we're done.

But hopefully that gives you a functional workaround.

[Marc013]

@BernieWhite, Thank you very much! I'll implement this solution.