PesterGroupResource

[SteveL-MSFT]

Summary of the new feature / enhancement

For audit scenarios, enable execution of Pester tests. A property of this resource would provide a path to the tests and recursively all *.test.ps1 would get executed. Need to consider what the output object looks like so that not only is it easy to tell if all the tests pass, but which individual tests failed with failure information.

Proposed technical implementation details (optional)

This might be a test only resource which currently isn't supported, but we can add a new property in the resource manifest or have this resource like the AssertionGroup and always execute "test" even if called by get and set.

[michaeltlombardi]

I think, if we standardize on some munged representation of the Pester.Test object as the resource surface, we could use the Pester configuration to run the tests in discovery mode for get and actually invoke the tests in set:

$PesterConfig = New-PesterConfiguration -Hashtable @{
    Run = @{
        TestExtension = '.dsc.tests.ps1'
        PassThru      = $true
        SkipRun       = $true
    }
}

$GetSuite = Invoke-Pester -Configuration $PesterConfig

Starting discovery in 1 files.
Discovery found 2 tests in 55ms.

Test run was skipped.
Tests completed in 54ms
Tests Passed: 0, Failed: 0, Skipped: 0 NotRun: 2

$GetSuite.Tests[0]

Name              : automatic_update_is_enabled
Path              : {tstoy, Config, Machine, automatic_update_is_enabled}
Data              : 
ExpandedName      : automatic_update_is_enabled
ExpandedPath      : tstoy.Config.Machine.automatic_update_is_enabled
Result            : NotRun
ErrorRecord       : {}
StandardOutput    : 
Duration          : 00:00:00
ItemType          : Test
Id                : 
ScriptBlock       : 
                                    $Config.Updates.Automatic | Should -BeTrue

Tag               : 
Focus             : False
Skip              : False
Block             : [ ] Machine
First             : True
Last              : False
Include           : False
Exclude           : False
Explicit          : False
ShouldRun         : True
StartLine         : 7
Executed          : False
ExecutedAt        : 
Passed            : False
Skipped           : False
UserDuration      : 00:00:00
FrameworkDuration : 00:00:00
PluginData        : 
FrameworkData     : 

The trick is going to be figuring out what makes sense to return as the representation for a test - Probably some munged object including ExpandedPath, Result, ErrorRecord, Duration, Data, ScriptBlock, StartLine, and Tag?

Then Get could show you the tests that will run and Test shows you the result of running them. I think for the best UX, users should only control where/how Pester runs with the top-level resource (in iteration one, no support for filtering tests or providing override data). The resource itself generates the array of dynamic resource representations to return for get and test.

I think we can get away with that, as long as we have a way for dsc to understand dynamic resource declarations for provider/group resources, or we make the Pester resource fully map to the normative output for get and test. If there was a way for a resource provider to define dynamic resources as part of the configuration processing step before an operation, that would be ideal.

So that would look something like defining this configuration:

# Authored example.dsc.config.yaml before processing
$schema: https://raw.githubusercontent.com/PowerShell/DSC/main/schemas/2023/10/config/document.json
resources:
- name: Pester Validation
  type: DSC/PesterSuite
  properties:
    testsFolderPath: /data/tests

Becoming this data model after the dynamic resource hook / config processing, before any further operations:

# Processed config
$schema: https://raw.githubusercontent.com/PowerShell/DSC/main/schemas/2023/10/config/document.json
resources:
- name: Pester Validation
  type: DSC.Pester/Suite
  properties:
    testsFolderPath: /data/tests
    # Representing the desired state
    resources:
      - type: DSC.Pester/Test
        name: tstoy.config.machine.automatic_update_is_enabled
        properties:
          # Technically, nearly all of these properties should be read-only from a schema
          # perspective - the only ones the resource should actually check are the errorRecord
          # and result. I'm not sure whether/how the provider could cache these during config
          # processing, but that's a future-iteration problem.
          filePath:    /data/tests/tstoy.dsc.tests.ps1
          startLine:   7
          scriptBlock: |
              $Config.Updates.Automatic | Should -BeTrue
          errorRecord: null
          result:      Passed
      - type: DSC.Pester/Test
        name: tstoy.config.machine.checks_updates_daily
        properties:
          filePath:    /data/tests/tstoy.dsc.tests.ps1
          startLine:   10
          scriptBlock: |
              $Config.Updates.CheckFrequency | Should -Be 1
          errorRecord: null
          result:      Passed

The Dsc.Pester/Suite provider should be able to very quickly return results for get, since it's just passing the dynamically created resource instances back to DSC.

For test, it would invoke the entire suite up front (or each file separately, whatever, there's a lot of options) and then populate the individual resources to return them.

Example output for `test`

```yaml $schema: https://raw.githubusercontent.com/PowerShell/DSC/main/schemas/2023/10/config/document.json resources: - name: Pester Validation type: DSC.Pester/Suite properties: testsFolderPath: /data/tests # Representing the desired state resources: - type: DSC.Pester/Test name: tstoy.config.machine.automatic_update_is_enabled desiredState: filePath: /data/tests/tstoy.dsc.tests.ps1 startLine: 7 scriptBlock: | $Config.Updates.Automatic | Should -BeTrue errorRecord: null result: Passed actualState: filePath: /data/tests/tstoy.dsc.tests.ps1 startLine: 7 scriptBlock: | $Config.Updates.Automatic | Should -BeTrue errorRecord: Expected $true, but got $false. result: Failed inDesiredState: false differingProperties: - errorRecord - result - type: DSC.Pester/Test name: tstoy.config.machine.checks_updates_daily desiredState: filePath: /data/tests/tstoy.dsc.tests.ps1 startLine: 10 scriptBlock: | $Config.Updates.CheckFrequency | Should -Be 1 errorRecord: null result: Passed actualState: filePath: /data/tests/tstoy.dsc.tests.ps1 startLine: 10 scriptBlock: | $Config.Updates.CheckFrequency | Should -Be 1 errorRecord: null result: Passed inDesiredState: true differingProperties: [] ```

If we can't dynamically define the resources (or as a first iteration), we could instead have users define a configuration document like:

# Authored example.dsc.config.yaml without pre-processing implemented
$schema: https://raw.githubusercontent.com/PowerShell/DSC/main/schemas/2023/10/config/document.json
resources:
- name: Pester Validation
  type: DSC.Pester/Suite
  properties:
    testsFolderPath: /data/tests
    # Representing the desired state
    resources:
      - type: DSC.Pester/Test
        name: tstoy.config.machine.automatic_update_is_enabled
        properties:
          filePath:    /data/tests/tstoy.dsc.tests.ps1
      - type: DSC.Pester/Test
        name: tstoy.config.machine.checks_updates_daily
        properties:
          filePath:    /data/tests/tstoy.dsc.tests.ps1

On dynamic resources from discovery

I think that one way to do this would be to mark a provider as supporting dynamic resource definitions in the manifest. This could actually be convenient for author providers of this kind, like inspec tests, or terraform files, etc. From an implementation perspective, DSC could always call these providers to dynamically populate their resources if that property isn't defined, and otherwise use the provider like any other if resources is explicitly defined.

$schema: https://raw.githubusercontent.com/PowerShell/DSC/main/schemas/2023/10/config/document.json
resources:
- name: Pester Validation with dynamic resources
  type: DSC.Pester/Suite
  properties:
    testsFolderPath: /data/tests
- name: Pester Validation with specified resources
  type: DSC.Pester/Suite
  properties:
    testsFolderPath: /data/website/tests
    resources:
      - type: DSC.Pester/Test
        name: frontend.home.serves_200
        properties:
          filePath: /data/website/tests/frontend.dsc.tests.ps1

[SteveL-MSFT]

Keep in mind that we want to define a pattern here that could apply to other similar sources like Chef InSpec, etc... which also means that we can have a specific property in the resource manifest to identify these class of providers if DSC needs to handle them slightly differently (output at least).