Error in ALOps Run TestSuite

[ps610]

Hello, for roughly more than a week we have been having problems running tests in some of our pipelines. Specifically, it is a BC14 project and we experience the following error during the "ALOps Run TestSuite" step: "Run-Test-Exception: Exception calling ".ctor" with "2" argument(s): "Value cannot be null.". We are using the modified test page (see #54) and had no issues with our pipelines for at least a year.

Our last successful pipeline run was on July 21 (vacation season :/ ) and it seems the errors started to appear at least on July 25.

last successfully run:

##[section]Starting: ALOps Run TestSuite
==============================================================================
Task         : ALOps App Test
Description  : Run AL-TestSuite for Business Central
Version      : 1.449.3024
Author       : Hodor
Help         : Run Business Central Test-Suite and collect results.
==============================================================================
*** Validate configuration
*** Task Inputs:

name                           value
----                           -----
usedocker                       True
fixed_tag                           
nav_serverinstance             BC140
tenant                       default
companyname                         
profile                             
bc_username                         
bc_password                         
bc_authentication    NavUserPassword
bc_webclient_url                    
testfilter              55500..55799
disabledtests                       
show_available_tests           False
import_testtoolkit             False
import_action              Overwrite
testpage                      130409
testsuite                    DEFAULT
extensionid                         
installaltesttool              False
failed_test_action           Warning
override_finsql_path                
resultfilename       TestResults.xml

*** For documentation, please visit   : https://www.alops.be/documentation

*** ALOps License:
  * Licensed To: Cosmo Consult - Azure Devops & Docker Self-Service

*** Importing required PS-Functions
*** Working folder: C:\azp\agent\_work\1\s
***
*** Prepare Docker Session
*** Connect Docker Session
*** Initiate Docker Session
*** Direct PSSession to other container [2468d7e512a78c8d2504813e2dbb31a0987e37e8341026427fd013f1eae59f8a]
*** Container resolved to hostname [2468d7e512a7]
*** Set Docker Container ErrorActionPreference = Stop
*** Copy Modules to Docker
*** Setup Docker Session
*** Checking for container-specific Test Assemblies
*** Using container-specific Test Assemblies
*** Import Modules in Docker Session
*** Import UI Client [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
VERBOSE: Loading module from path 'C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll'.
*** Import NewtonSoft JSON [C:\Program Files\Microsoft Dynamics NAV\140\Service\Newtonsoft.Json.dll]
*** Import Modules in Docker Session: class_ClientContext
*** Test Context Url: https://2468d7e512a7/NAV/cs?tenant=default
*** Setup BC Credentials in Docker
*** Resolve Docker Credentials
*** Set TCP Keep-Alive
*** Start TestSuite
*** Running TestSuite with authentication [NavUserPassword] with User [admin]
*** Run warm-up with page [130409]
*** Warm-up Client Context
*** Create Client Context
*** Add Tests to Default-TestSuite with filter [55500..55799]
Show form {0001FD69-0000-0000-0108-0000836BD2D2}
Title: PS Test Tool Page Edit - Test Tool - DEFAULT
Apply filter for Test-Codeunits: [55500..55799]
*** Run Available Tests for suite [DEFAULT] with page [130409]
*** Connect TestClient with page [130409]
Show form {0001FD69-0000-0000-0108-0000836BD2D2}
Title: PS Test Tool Page Edit - Test Tool - DEFAULT
  Codeunits XXX ...
True
*** XUnit Test results: c:\Run\DevOps\TestResults.xml
<?xml version="1.0" encoding="UTF-8"?>
<assemblies>
....
</assemblies>

*** Get XUnit Test file from [c:\Run\DevOps\TestResults.xml] to [C:\azp\agent\_work\1\s\TestResults.xml]

*** Disconnect Docker session.
*** Docker session removed.
*** Running Tests Completed.
*** Cleanup VSTS Environment: True
##[section]Finishing: ALOps Run TestSuite

failed run:

##[section]Starting: ALOps Run TestSuite
==============================================================================
Task         : ALOps App Test
Description  : Run AL-TestSuite for Business Central
Version      : 1.451.3127
Author       : Hodor
Help         : Run Business Central Test-Suite and collect results.
==============================================================================
*** Validate configuration
*** Task Inputs:

name                           value
----                           -----
usedocker                       True
fixed_tag                           
nav_serverinstance             BC140
tenant                       default
companyname                         
profile                             
bc_username                         
bc_password                         
bc_authentication    NavUserPassword
bc_webclient_url                    
testfilter              55500..55799
disabledtests                       
show_available_tests           False
import_testtoolkit             False
import_action              Overwrite
testpage                      130409
testsuite                    DEFAULT
extensionid                         
installaltesttool              False
failed_test_action           Warning
override_finsql_path                
resultfilename       TestResults.xml
testrunnercodeunitid                

*** For documentation, please visit   : https://www.alops.be/documentation

*** ALOps License:
  * Licensed To: Cosmo Consult - Azure Devops & Docker Self-Service

*** Importing required PS-Functions
*** Working folder: C:\azp\agent\_work\1\s
***
*** Prepare Docker Session
*** Connect Docker Session
*** Initiate Docker Session
*** Direct PSSession to other container [22a24f7d8c4cda7c12819f0a3ac50e1063b5369ec23339ae385046d83277cce8]
*** Container resolved to hostname [22a24f7d8c4c]
*** Set Docker Container ErrorActionPreference = Stop
*** Copy Modules to Docker
*** Setup Docker Session
*** Checking for container-specific Test Assemblies
*** Using container-specific Test Assemblies
*** Import Modules in Docker Session
*** Import UI Client [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
VERBOSE: Loading module from path 'C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll'.
*** Import NewtonSoft JSON [C:\Program Files\Microsoft Dynamics NAV\140\Service\Newtonsoft.Json.dll]
*** Import Modules in Docker Session: class_ClientContext
*** Test Context Url: https://22a24f7d8c4c/NAV/cs?tenant=default
*** Setup BC Credentials in Docker
*** Resolve Docker Credentials
*** Set TCP Keep-Alive
*** Start TestSuite
*** Running TestSuite with authentication [NavUserPassword] with User [admin]
*** Run warm-up with page [130409]
*** Warm-up Client Context
*** Create Client Context
*** Add Tests to Default-TestSuite with filter [55500..55799]
Show form {0001FD69-0000-0000-0108-0000836BD2D2}
Title: PS Test Tool Page Edit - Test Tool - DEFAULT
Apply filter for Test-Codeunits: [55500..55799]
##[error]Run-Test-Exception: Exception calling ".ctor" with "2" argument(s): "Value cannot be null.
Parameter name: logicalControl"
*** Get XUnit Test file from [] to [C:\azp\agent\_work\1\s\]
##[error]Cannot bind argument to parameter 'Path' because it is null.
##[section]Finishing: ALOps Run TestSuite

Expected behavior The tests should be executed as in the past.

Thank you

[ps610]

Hello, any update on this?

[ps610]

The error still occurs and we are currently unable to run tests within pipelines.

[waldo1001]

Sorry to hear that. As requested while creating an issue: can you provide the complete yaml, please?

[ps610]

@waldo1001 This is the yaml of the Test Step:

# ...
variables:
  - name: 'job.Test.TestPage'
    value: '130409'
  - name: 'job.Test.Filter'
    value: $[ coalesce(variables['Test.Filter'], '50000..99000') ] # 'Test.Filter' set to '55500..55799'
  - name: 'job.Test.TestSuite'
    value: $[ coalesce(variables['Test.TestSuite'], 'DEFAULT') ]
  - name:  'job.Test.FailAction'
    value: $[ coalesce(variables['Test.FailAction'], 'Warning') ]
  - name:  'job.Test.FailPublishOnFailedTest'
    value: $[ coalesce(variables['Test.FailPublishOnFailedTest'], 'true') ]
# ...

# ...
parameters:
  displayName:    'Run TestSuite'
  condition:      succeeded()

steps:
- task: ALOpsAppTest@1
  displayName:            'ALOps ${{ parameters.displayName }}'
  condition:              ${{ parameters.condition }}
  inputs:
    usedocker:            true
    show_available_tests: false
    import_testtoolkit:   false
    testpage:             '$(job.Test.TestPage)'
    testfilter:           '$(job.Test.Filter)'
    testsuite:            '$(job.Test.TestSuite)'
    failed_test_action:   '$(job.Test.FailAction)'

- task: PublishTestResults@2
  condition:                ${{ parameters.condition }}
  displayName:              '${{ parameters.displayName }}'
  inputs:
    testResultsFormat:      XUnit
    testResultsFiles:       '**/TestResults.xml'
    testRunTitle:           'BC Test Results: $(Build.BuildId)'
    failTaskOnFailedTests:  '$(job.Test.FailPublishOnFailedTest)'
# ...

Does this help?

[waldo1001]

Well, no. If I don't see actual values, it's difficult to know what is going on. What about the full output?

[ps610]

Well, I guess the problem was on our end. I tried to reproduce the error in a less complex environment for illustration and noticed that the tests were executed there. Even after adding an existing database backup in the pipeline (with the modified test page) the tests were executed there.

I imported the test page again in our currently used database backup and now the tests are running again.. I am afraid one of my colleagues updated the database backup and used an older version of the test page.

Apologies for the inconvenience.

[waldo1001]

Problem solves is all I care. ;-). Sorry that I was unresponsive - this issue got lost, and I didn't really have much to look at ...