search

sam-cogan / arm-ttk-extension-xplatform

MIT License
11 stars 4 forks source link

When using a comma separated lists for -skiptest get a parameter 'clientSecret' error #27

Open bm-fez opened 7 months ago

bm-fez commented 7 months ago

Issue

I have just migrated from the Windows PowerShell to this Cross Platform version. The only change I thought we required was in the task name swapping to

- task: RunARMTTKTestsXPlat@1
  displayName: "Run ARM TTK tests on built ARM"
  inputs:
    templatelocation: '$(Build.SourcesDirectory)\BicepTemplates\ARMoutputs\*.json'
    resultLocation: '$(System.DefaultWorkingDirectory)\results'
    skipTests: '$(TestsToSkip)'
    cliOutputResults: true

However I have found that if have a comma separated list for -skipTests we get the following error

Using executable 'powershell.exe' powershell.exe E:\Agent_work_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.1\powershell\ps-runner.ps1 -templateLocation E:\Agent_work\17\s\BicepTemplates\ARMoutputs*.json -resultLocation E:\Agent_work\17\s\results -skipTests VM Images Should Use Latest Version,Resources Should Have Location,Location Should Not Be Hardcoded,Template Should Not Contain Blanks,DeploymentTemplate Must Not Contain Hardcoded Uri,Outputs Must Not Contain Secrets,URIs Should Be Properly Constructed -cliOutputResults -recurse

[error]E:\Agent_work_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.1\powershell\ps-runner.ps1 : Cannot

[error]E:\Agent_work_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.1\powershell\ps-runner.ps1 : Cannot

[error]process argument transformation on parameter 'clientSecret'. Cannot convert value to type System.String.

At line:1 char:265 ... ults -skipTests VM Images Should Use Latest Version,Resources Should ... CategoryInfo : InvalidData: (:) [ps-runner.ps1], ParameterBindingArgumentTransformationException FullyQualifiedErrorId : ParameterArgumentTransformationError,ps-runner.ps1

Analysis

Experimentation of running the ps-runner.ps1 from the command prompt on a self hosted build agent showed that the solution was to enclose the skip test list for the inner PowerShell running in quotes.

powershell\ps-runner.ps1 -templateLocation E:\Agent\_work\17\s\BicepTemplates\ARMoutputs\*.json -resultLocation E:\Agent\_work\17\s\results -skipTests 'VM Images Should Use Latest Version,Resources Should Have Location,Location Should Not Be Hardcoded,Template Should Not Contain Blanks,DeploymentTemplate Must Not Contain Hardcoded Uri,Outputs Must Not Contain Secrets,URIs Should Be Properly Constructed' -cliOutputResults -recurse

Once this was done ARM TTK ran as expected.

Have I missed something related to parameter passing, am I passing in the list incorrectly?

I will link a PR with this change to this issue on the assumption this fix is valid. I have assumed the same fix is needed for the includeTests parameter

sam-cogan commented 7 months ago

Hi Richard, I can't see anything you've done wrong here, looks like it is a bug. Thanks very much for the PR, will get that merged now.

bm-fez commented 7 months ago

@sam-cogan I can see that the #28 has been merged and there is a new release 1.2.2 in GitHub, but the new release is not showing in the Azure DevOps Marketplace.

When I check using the REST API, in a browser I issue https://[dev.azure.com//_apis/distributedtask/tasks/cb27e003-8534-4da9-aaa0-35ca8756ae29 it is still showing the 1.2.1 as being the only version of the task available to my organisation.

image

Just wondering if a publish step has failed for the extension?

sam-cogan commented 7 months ago

Hi Richard, sorry that was my fault, I haven't automated the bumping of the version in the task.json, so it didn't pickup the change. This should be sorted now.

bm-fez commented 7 months ago

I can confirm it is working now - my pipelines are failing with out of date Azure resources just like I expected

Thanks

rob-hendrickx-c4t commented 7 months ago

@sam-cogan I actually observe failures on our pipeline starting from the moment version 1.2.2 was used. In our case the pipeline uses an ubuntu agent. We don't get a "hard error", but the tests we provided to the skipTests parameter as a comma separated list are no longer skipped. image Both with and without quotes in our yaml file, it fails with the same result. image

We'll revert to version 1.2.1 (were I can confirm everything works as expected) until this is fixed for all scenarios

sam-cogan commented 7 months ago

Ok, looks like there is some sort of inconsistency here. @rfennell is your build agent Windows or Linux? I'm wondering if that is part of the problem, given Rob is using Linux

bm-fez commented 7 months ago

Most of our agents are on prem running Windows, but I have just swapped on of my builds that uses this task to run in ubuntu-latest, and it has run without error 

        - task: RunARMTTKTestsXPlat@1
          displayName: "Run ARM TTK tests on built ARM"
          inputs:
            templatelocation: '$(Build.SourcesDirectory)/BicepTemplates/ARMoutputs/*.json'
            resultLocation: '$(System.DefaultWorkingDirectory)/results'
            skipTests: VM Images Should Use Latest Version,Resources Should Have Location,Location Should Not Be Hardcoded,Template Should Not Contain Blanks,DeploymentTemplate Must Not Contain Hardcoded Uri,Outputs Must Not Contain Secrets, URIs Should Be Properly Constructed
            cliOutputResults: true

Returns

2024-01-15T17:15:24.1748025Z ##[debug]Evaluating condition for step: 'Run ARM TTK tests on built ARM'
2024-01-15T17:15:24.1748628Z ##[debug]Evaluating: SucceededNode()
2024-01-15T17:15:24.1748836Z ##[debug]Evaluating SucceededNode:
2024-01-15T17:15:24.1749164Z ##[debug]=> True
2024-01-15T17:15:24.1749364Z ##[debug]Result: True
2024-01-15T17:15:24.1749598Z ##[section]Starting: Run ARM TTK tests on built ARM
2024-01-15T17:15:24.1753155Z ==============================================================================
2024-01-15T17:15:24.1753265Z Task         : Run Azure RM TTK Tests (Cross Platform)
2024-01-15T17:15:24.1753340Z Description  : Run Azure Resource Manager Template Test Kit Tests against ARM & Bicep Templates
2024-01-15T17:15:24.1753462Z Version      : 1.2.2
2024-01-15T17:15:24.1753515Z Author       : Sam Cogan
2024-01-15T17:15:24.1753570Z Help         : [More Information](https://github.com/sam-cogan/arm-ttk-extension/blob/master/README.md)
2024-01-15T17:15:24.1753684Z ==============================================================================
2024-01-15T17:15:24.2500193Z ##[debug]Using node path: /home/vsts/agents/3.232.1/externals/node10/bin/node
2024-01-15T17:15:24.5024864Z ##[debug]agent.TempDirectory=/home/vsts/work/_temp
2024-01-15T17:15:24.5051108Z ##[debug]loading inputs and endpoints
2024-01-15T17:15:24.5068985Z ##[debug]loading INPUT_TEMPLATELOCATION
2024-01-15T17:15:24.5087331Z ##[debug]loading INPUT_RECURSE
2024-01-15T17:15:24.5092480Z ##[debug]loading INPUT_RESULTLOCATION
2024-01-15T17:15:24.5096773Z ##[debug]loading INPUT_SKIPTESTS
2024-01-15T17:15:24.5100810Z ##[debug]loading INPUT_ALLTEMPLATESMAIN
2024-01-15T17:15:24.5104701Z ##[debug]loading INPUT_CLIOUTPUTRESULTS
2024-01-15T17:15:24.5108534Z ##[debug]loading INPUT_IGNOREEXITCODE
2024-01-15T17:15:24.5112383Z ##[debug]loading INPUT_USEPSCORE
2024-01-15T17:15:24.5116175Z ##[debug]loading INPUT_USEAZBICEP
2024-01-15T17:15:24.5120342Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2024-01-15T17:15:24.5124651Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2024-01-15T17:15:24.5128963Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2024-01-15T17:15:24.5137162Z ##[debug]loading SECRET_SNIPEITMYSQLPASSWORD
2024-01-15T17:15:24.5141361Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
2024-01-15T17:15:24.5145499Z ##[debug]loaded 14
2024-01-15T17:15:24.5153763Z ##[debug]Agent.ProxyUrl=undefined
2024-01-15T17:15:24.5157681Z ##[debug]Agent.CAInfo=undefined
2024-01-15T17:15:24.5160681Z ##[debug]Agent.ClientCert=undefined
2024-01-15T17:15:24.5163774Z ##[debug]Agent.SkipCertValidation=undefined
2024-01-15T17:15:24.5181105Z ##[debug]templateLocation=/home/vsts/work/1/s/BicepTemplates/ARMoutputs/*.json
2024-01-15T17:15:24.5185420Z ##[debug]resultLocation=/home/vsts/work/1/s/results
2024-01-15T17:15:24.5188413Z ##[debug]includeTests=undefined
2024-01-15T17:15:24.5192362Z ##[debug]skipTests=VM Images Should Use Latest Version,Resources Should Have Location,Location Should Not Be Hardcoded,Template Should Not Contain Blanks,DeploymentTemplate Must Not Contain Hardcoded Uri,Outputs Must Not Contain Secrets, URIs Should Be Properly Constructed
2024-01-15T17:15:24.5195535Z ##[debug]mainTemplates=undefined
2024-01-15T17:15:24.5198803Z ##[debug]allTemplatesAreMain=undefined
2024-01-15T17:15:24.5205418Z ##[debug]cliOutputResults=true
2024-01-15T17:15:24.5209313Z ##[debug]ignoreExitCode=false
2024-01-15T17:15:24.5213024Z ##[debug]recurse=true
2024-01-15T17:15:24.5215931Z ##[debug]azureServiceConnection=undefined
2024-01-15T17:15:24.5219534Z ##[debug]useAzBicep=false
2024-01-15T17:15:24.5222428Z ##[debug]System.Debug=True
2024-01-15T17:15:24.5225208Z ##[debug]AGENT.OS=Linux
2024-01-15T17:15:24.5228464Z ##[debug]AGENT.OS=Linux
2024-01-15T17:15:24.5235641Z Using executable 'pwsh' as only only option on 'Linux'
2024-01-15T17:15:24.5241348Z pwsh /home/vsts/work/_tasks/RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29/1.2.2\powershell\ps-runner.ps1 -templateLocation /home/vsts/work/1/s/BicepTemplates/ARMoutputs/*.json -resultLocation /home/vsts/work/1/s/results -skipTests 'VM Images Should Use Latest Version,Resources Should Have Location,Location Should Not Be Hardcoded,Template Should Not Contain Blanks,DeploymentTemplate Must Not Contain Hardcoded Uri,Outputs Must Not Contain Secrets, URIs Should Be Properly Constructed' -cliOutputResults -recurse
2024-01-15T17:15:26.9363400Z A newer version of the ARM-TTK is available at: https://github.com/Azure/arm-ttk/releases
2024-01-15T17:15:26.9363964Z 
2024-01-15T17:15:26.9364645Z Installed Version: 0.23
2024-01-15T17:15:26.9364759Z 
2024-01-15T17:15:26.9364873Z Latest Version: 0.24
2024-01-15T17:15:26.9364976Z 
2024-01-15T17:15:28.6344047Z ##[debug]Agent running environment resource - Disk: available:31256.00MB out of 85160.00MB, Memory: used 25MB out of 6921MB, CPU: usage 31.71
2024-01-15T17:15:29.2744063Z ##[section] /home/vsts/work/1/s/BicepTemplates/ARMoutputs/CoreDeployment.json passed all tests
2024-01-15T17:15:29.2744638Z 
2024-01-15T17:15:29.3608747Z Script finished
2024-01-15T17:15:29.3638763Z ##[section]Finishing: Run ARM TTK tests on built ARM

I got the same results when

@rob-hendrickx-c4t what does your Azure DevOps log show for the generated ps-runner.ps1 command line?

mbraekman commented 7 months ago

As mentioned by @rob-hendrickx-c4t we're experiencing the same issues in different projects when using version 1.2.2 of the ARMTTK-task, as it appears the skipTests-parameter is no longer being taken into account. Even though several tests have been marked to be skipped, these are not considered.

Running ARM-TTK verification separately on my machine does work as expected.

Details of our pipelines:

The following configuration works as expected:

  - task: RunARMTTKTestsXPlat@1.2.1
    displayName: "Run ARM TTK tests"
    inputs:
      templatelocation: "$(System.DefaultWorkingDirectory)/$(rootDirectory)$(projectName).Deployment/Bicep/"
      resultLocation: "$(System.DefaultWorkingDirectory)/results"
      skipTests: |
        apiversions should be recent,
        apiVersions Should Be Recent In Reference Functions,
        DeploymentTemplate Must Not Contain Hardcoded Uri,
        IDs Should Be Derived From ResourceIDs,
        Template Should Not Contain Blanks,
        URIs Should Be Properly Constructed,
        Parameter Types Should Be Consistent
      usePsCore: true

This doesn't work (autom. resolves to version 1.2.2) and fails against several test cases provided in the skipTests-parameter:

  - task: RunARMTTKTestsXPlat@1
    displayName: "Run ARM TTK tests"
    inputs:
      templatelocation: "$(System.DefaultWorkingDirectory)/$(rootDirectory)$(projectName).Deployment/Bicep/"
      resultLocation: "$(System.DefaultWorkingDirectory)/results"
      skipTests: |
        apiversions should be recent,
        apiVersions Should Be Recent In Reference Functions,
        DeploymentTemplate Must Not Contain Hardcoded Uri,
        IDs Should Be Derived From ResourceIDs,
        Template Should Not Contain Blanks,
        URIs Should Be Properly Constructed,
        Parameter Types Should Be Consistent
      usePsCore: true

One of the resulting errors:

2024-01-15T15:00:39.3270433Z ##[section]Starting: Run ARM TTK tests
2024-01-15T15:00:39.3422106Z ==============================================================================
2024-01-15T15:00:39.3422313Z Task         : Run Azure RM TTK Tests (Cross Platform)
2024-01-15T15:00:39.3422428Z Description  : Run Azure Resource Manager Template Test Kit Tests against ARM & Bicep Templates
2024-01-15T15:00:39.3422580Z Version      : 1.2.2
2024-01-15T15:00:39.3422685Z Author       : Sam Cogan
2024-01-15T15:00:39.3422770Z Help         : [More Information](https://github.com/sam-cogan/arm-ttk-extension/blob/master/README.md)
2024-01-15T15:00:39.3422928Z ==============================================================================
2024-01-15T15:00:40.8226786Z Using executable 'pwsh'
2024-01-15T15:00:40.8227576Z pwsh D:\a\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.2\powershell\ps-runner.ps1 -templateLocation D:\a\1\s/src/MO.Document.Gateway.CA.PO.Deployment/Bicep/ -resultLocation D:\a\1\s/results -skipTests 'apiversions should be recent,
2024-01-15T15:00:40.8228145Z apiVersions Should Be Recent In Reference Functions,
2024-01-15T15:00:40.8228392Z DeploymentTemplate Must Not Contain Hardcoded Uri,
2024-01-15T15:00:40.8228631Z IDs Should Be Derived From ResourceIDs,
2024-01-15T15:00:40.8228858Z Template Should Not Contain Blanks,
2024-01-15T15:00:40.8229079Z URIs Should Be Properly Constructed,
2024-01-15T15:00:40.8229404Z Parameter Types Should Be Consistent' -recurse
2024-01-15T15:00:42.8238004Z A newer version of the ARM-TTK is available at: https://github.com/Azure/arm-ttk/releases
2024-01-15T15:00:42.8238230Z 
2024-01-15T15:00:42.8238402Z Installed Version: 0.23
2024-01-15T15:00:42.8238495Z 
2024-01-15T15:00:42.8238656Z Latest Version: 0.24
2024-01-15T15:00:42.8238740Z 
2024-01-15T15:03:16.5119097Z ##[error]Exception: Failures found in test results


2024-01-15T15:03:16.5131751Z ##[error]Exception: Failures found in test results


2024-01-15T15:03:16.5882246Z Script finished
2024-01-15T15:03:16.6638724Z ##[section]Finishing: Run ARM TTK tests
Microsoft.PowerShell.Commands.WriteErrorException: Api versions must be the latest or under 2 years old (730 days) - API version 2021-05-01 of Microsoft.Network/privateEndpoints is 989 days old Microsoft.PowerShell.Commands.WriteErrorException: Api versions must be the latest or under 2 years old (730 days) - API version 2021-05-01 of Microsoft.Network/privateEndpoints/privateDnsZoneGroups is 989 days old in template file sqlserver-privateendpoint.json

As a current workaround, we've reverted back to using v1.2.1, until this has been resolved.

sam-cogan commented 7 months ago

Ok, in the short term, I've reverted the changer, so if you switch to latest, or use 1.2.3 then it should be back to the way it was, whilst I find time to investigate the actual problem,

@rfennell it would be interesting to know if you see the original issue when using the usePSCore flag? If it works fine then it may be only a PS 3/4 issue.

rob-hendrickx-c4t commented 7 months ago

@rob-hendrickx-c4t what does your Azure DevOps log show for the generated ps-runner.ps1 command line?

In our case it shows the following on version 1.2.2 (which doesn't work as expected for us)

pwsh /home/vsts/work/_tasks/RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29/1.2.2\powershell\ps-runner.ps1 -templateLocation /home/vsts/work/1/s/Subscription -resultLocation /home/vsts/work/1/s/results -skipTests 'Template-Should-Not-Contain-Blanks,apiVersions-Should-Be-Recent' -mainTemplates Main.bicep -cliOutputResults -recurse

and this on version 1.2.1 (which works as expected)

pwsh /home/vsts/work/_tasks/RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29/1.2.1\powershell\ps-runner.ps1 -templateLocation /home/vsts/work/1/s/Subscription -resultLocation /home/vsts/work/1/s/results -skipTests Template-Should-Not-Contain-Blanks,apiVersions-Should-Be-Recent -mainTemplates Main.bicep -cliOutputResults -recurse

One observation that might help @bm-fez: we specify the test names with dashes instead of spaces, which removes the need to wrap the tests in quotes altogether. This by the way also seems what the documentation refers to https://github.com/Azure/arm-ttk/tree/master/arm-ttk/testcases/deploymentTemplate image

EDIT: I just notice that @mbraekman also uses spaces and it didn't cause issues earlier, so nvm my last remark

bm-fez commented 7 months ago

@sam-cogan interestingly with the reverted 1.2.3 when using usePSCore=true I don't see the problem.

So for me that is a good enough fix i.e use PWSH to execute ARM TTK

sam-cogan commented 7 months ago

Ok, that's great to hear. I will look into why older PowerShell has issues, but it's a lower priority now you've got a solution.

bm-fez commented 7 months ago

I think the issue is related to #30 the issue I opened and closed today. It is all down to the interaction between the Azure DevOps Node execution engine and Powershell.exe not being the same as for pwsh.exe.

I suspect the answer is to add the quotes around that skip and include parameters only when powershell.exe is being used