In-flight validation failing

[smholvoet]

Description of the issue

I'm trying to run in-flight validation of already deployed resources using Azure Pipelines.

To Reproduce

Steps to reproduce the issue:

Below is the stage which runs the tests.

• I first install PSRule and the PSRule.Rules.Azure modules
• I then use Azure PowerShell to connect to my subscription and run the actual validation

- stage: SmokeTest
  jobs:
  - job: SmokeTest
    displayName: Smoke test
    steps:
      - task: ps-rule-install@0
        displayName: Install PSRule.Rules.Azure
        inputs:
          module: 'PSRule.Rules.Azure'

      - task: AzurePowerShell@5
        inputs:
          azureSubscription: 'my.secret.subscription'
          ScriptType: 'InlineScript'
          Inline: |
            Get-AzContext;
            Export-AzRuleData -OutputPath 'out/templates/';
            Assert-PSRule -Module 'PSRule.Rules.Azure' -InputPath 'out/templates/';
          azurePowerShellVersion: 'LatestVersion'

Expected behaviour

Expecting a list of results, either failed or passed

   TargetName: storage

RuleName                            Outcome    Recommendation
--------                            -------    --------------
Azure.Storage.UseReplication        Fail       Storage accounts not using GRS may be at risk
Azure.Storage.SecureTransferRequ... Fail       Storage accounts should only accept secure traffic
Azure.Storage.SoftDelete            Fail       Enable soft delete on Storage Accounts

Error output

Starting: AzurePowerShell
==============================================================================
Task         : Azure PowerShell
Description  : Run a PowerShell script within an Azure environment
Version      : 5.185.0
Author       : Microsoft Corporation
Help         : https://aka.ms/azurepowershelltroubleshooting
==============================================================================
Generating script.
/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command . '/home/vsts/work/_temp/24557fd4-f72a-4833-9302-dc3ac648989a.ps1'
Saved!
Import-Module -Name /home/vsts/.local/share/powershell/Modules/Az.Accounts/2.6.0/Az.Accounts.psd1 -Global
Clear-AzContext -Scope Process
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Connect-AzAccount -ServicePrincipal -Tenant *** -Credential System.Management.Automation.PSCredential -Environment AzureCloud @processScope
 Set-AzContext -SubscriptionId ... -TenantId ***

Name                                     Account   Subscript Environme TenantId
                                                   ionName   nt
----                                     -------   --------- --------- --------
my.secret.subscription      (z4x212y8-5e… z1x8y5c4… my.secre… AzureClo… a17bcde…
Get-AzRoleAssignment: /home/vsts/.local/share/powershell/Modules/PSRule.Rules.Azure/1.9.0/PSRule.Rules.Azure.psm1:1137
Line |
1137 |  … esources += Get-AzRoleAssignment -DefaultProfile $Context -Scope $Res …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception of type 'Microsoft.Rest.Azure.CloudException' was
     | thrown.

##[error]PowerShell exited with code '1'.
Finishing: AzurePowerShell

Module in use and version:

• Azure Devops extension: 1.3.0
• PSRule.Rules.Azure: 1.9.0
• PSRule: 1.8.0

Captured output from $PSVersionTable:

Azure PowerShell version: 5.185.0

Running this on the ubuntu-latest Microsoft-hosted agent.

[BernieWhite]

@smholvoet Thanks for reporting this. Looks like there is an issue with exporting role assignments from the subscription.

Are you able to confirm if you have any role assignments reporting as "unknown" within the Azure Portal?

Also are you able to confirm that the Azure DevOps Service Connection has at least Reader permissions against the subscription. If not, what Azure role is assigned.

[smholvoet]

I've created a service principal which has the Contributor role against the subscription. The service connection to Azure DevOps seems to be functioning OK as I've already set up a coupe of other stages in my IaC deployment pipeline. Previewing changes via az deployment group what-if for example is able to query existing resources within my subscription.

The app registration itself seems to have the Cloud application administrator role (set under Manage > Roles and administrators).

[BernieWhite]

Are you able to confirm if you have any role assignments reporting as "unknown" within the Azure Portal?

Some version of the Get-AzRoleAssignment will fail if a role assignment can not be read. This can occur if the role assignment exists but the Azure AD identity have been deleted. This can create role assignments that report as unknown.

[smholvoet]

Just had another look, it's not showing up as "unknown":

[BernieWhite]

Ok thanks. I'll see if I can reproduce the error. If possible, can you insert a new line Get-Error; after Export-AzRuleData -OutputPath 'out/templates/'; and provide the stack trace.

[smholvoet]

Appreciate the help.

Adding Get-Error; somehow didn't provide any additional error information. I'm just getting the original error. I ran the pipeline with the system diagnostics flag enabled, which changes the verbosity of all commands. Here's the export:

VERBOSE: [Export-AzRuleData] BEGIN::
VERBOSE: [Context] -- Found (1) subscription contexts
VERBOSE: [Context] -- Using subscription: my.secret.subscription
VERBOSE: [Context] -- Using [1/1] subscription contexts
VERBOSE: Performing the operation "out/templates/" on target "Create output directory".
VERBOSE: [Export] -- Using subscription: my.secret.subscription
VERBOSE: [Export] -- Getting Azure resources
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Compute/virtualMachines/VIRTUALMACHINE123/extensions/AzurePolicyforLinux
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/AzureBackupRG_westeurope_1/providers/Microsoft.Compute/restorePointCollections/AzureBackup_VIRTUALMACHINE123_1837521425428905810
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/NetworkWatcherRG/providers/Microsoft.Network/networkWatchers/NetworkWatcher_westeurope
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Compute/disks/VIRTUALMACHINE123_DataDisk_0
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Compute/disks/VIRTUALMACHINE123_OsDisk_1_9981ed4f56604ef1a34550dd6c5934e5
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Compute/virtualMachines/VIRTUALMACHINE123
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Compute/virtualMachines/VIRTUALMACHINE123/extensions/AzureNetworkWatcherExtension
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Compute/virtualMachines/VIRTUALMACHINE123/extensions/LinuxDiagnostic
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Compute/virtualMachines/VIRTUALMACHINE123/extensions/DependencyAgentLinux
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Network/networkInterfaces/VIRTUALMACHINE12396
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Compute/virtualMachines/VIRTUALMACHINE123/extensions/OmsAgentForLinux
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Network/networkSecurityGroups/VIRTUALMACHINE123-nsg
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group/providers/Microsoft.Network/virtualNetworks/my_vnet
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group-infra/providers/Microsoft.Storage/storageAccounts/mydiag
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group-infra/providers/Microsoft.RecoveryServices/vaults/my-resource-group-rsv
VERBOSE: [Export] -- Expanding: /subscriptions/<subscription-guid>/resourceGroups/my-resource-group
Get-AzRoleAssignment: /home/vsts/.local/share/powershell/Modules/PSRule.Rules.Azure/1.9.0/PSRule.Rules.Azure.psm1:1137
Line |
1137 |  … esources += Get-AzRoleAssignment -DefaultProfile $Context -Scope $Res …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception of type 'Microsoft.Rest.Azure.CloudException' was
     | thrown.

##[debug]Exit code 1 received from tool '/usr/bin/pwsh'
##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'
##[debug]task result: Failed
##[error]PowerShell exited with code '1'.
##[debug]Processed: ##vso[task.issue type=error;]PowerShell exited with code '1'.
##[debug]Processed: ##vso[task.complete result=Failed;]PowerShell exited with code '1'.

Looks like it is able to export some resources, but suddenly cuts out, even though I should be able to export all resources within the entire subscription...

[BernieWhite]

@smholvoet We made some significant changes to in-flight exports with release v1.24.0 to address other limitations such as #1341. This should also address the issue you were experiencing.

Can you please test to make that this is working as expected.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within 7 days. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has not had any recent activity.