Unable to Run Test Engine in ADO Pipeline

[jimiryquai]

I have tried to attach but the upload won't accept files with those extensions:

1. new_manyscreens_7720b_DocumentUri.msapp
2. testPlan.fx.yaml

Both of these are from here: manyscreens.

Describe the bug

I am trying to automate running test engine in ADO via PAC CLI (installed with Power Platform Tool installer Task) as follows:

YAML that passes parameters and creates env vars

 - task: PowerShell@2
      displayName: 'Run Test Engine '
      name: runTestEngine
      env:
        USER_EMAIL: $(serviceUser)
        USER_PASSWORD: $(servicePassword)
      inputs:
        filePath: '${{ parameters.buildSourcesDirectory }}\PowerPlatform\Tools\pac-tests-run.ps1'
        arguments: '-testPlanFile ''${{ parameters.buildSourcesDirectory }}\PowerPlatform\Tests\testPlan.fx.yml'' -environmentId ${{ parameters.environmentId }} -tenantId ${{ parameters.tenantId }} -outputDirectory ''${{ parameters.buildSourcesDirectory }}\PowerPlatform\Tests'''

Powershell that calls pac tests run

 param (
    [Parameter(Mandatory)] [String]$testPlanFile,
    [Parameter(Mandatory)] [String]$environmentId,
    [Parameter(Mandatory)] [String]$tenantId,
    [Parameter(Mandatory)] [String]$outputDirectory
)
    $pacPath = $env:POWERPLATFORMTOOLS_PACCLIPATH + "\pac\tools\pac.exe"
    $env:PATH = $env:PATH + ";" + $pacPath #note semi-colon delimeter in path for windows
    $pacCommand = "tests run -test $testPlanFile -env $environmentId -t $tenantId -o $outputDirectory"

    Invoke-Expression -Command "$pacPath $pacCommand"

environmentVariables consumed in testPlan.fx.yml

environmentVariables:
  users:
    - personaName: User1
      emailKey: $USER_EMAIL
      passwordKey: $USER_PASSWORD

To Reproduce

Steps to reproduce the behavior:

1. Run the pipeline

Expected Behavior:

The tests should run

Actual Behavior:

Chromium is installed but no tests are run and the Task times out (see image)

[jimiryquai]

I changed my environment variable yaml to the below as it shows in the docs:

environmentVariables:
   - users:
      - personaName: User1
        emailKey: USER_EMAIL
        passwordKey: USER_PASSWORD

I then got an actual error from the test engine task in my pipeline which is an improvement, in terms of actually getting feedback from the test, I guess:

[jt000]

Hi @jimiryquai, your correct that the values in the users for the emailKey and passwordKey should be the name of the environment variables & not the email\passwords themselves. I believe these should match the values set in the task env property as you have them. We hope to make login failures a little more recognizable in the future to help others out with similar issues (you're not the first to do this & I doubt you'll be the last :) ).

It sounds like you've resolved the issue you originally reported. I'm going to close this issue, but please feel free to reopen if I'm mistaken or open a new issue if you are seeing something new that you could use assistance with.

[jimiryquai]

Hey @jt000 I haven't resolved the issue - changing my yaml gave me some actual output but the output was that my yaml was invalid. As soon as I fixed that I went back to the problem of the pipeline running for 60 mins before timing out. I have tested the user creds locally and they are correct. Can we reopen this?