ALOpsAppTest doesn't work for BC16.3 and BC17.0 anymore

[DanielGoehler]

Describe the bug In BC 16.3 and BC17.0 ALOpsAppTest doesn't work. We get the error message Response status code does not indicate success: 500 (Internal Server Error).

*** AL-TestTool installation finished.
*** Set TCP Keep-Alive
*** Start TestSuite
*** Running TestSuite with authentication [NavUserPassword] with User [admin]
*** Run warm-up with page [130455]
*** Warm-up Client Context
CommunicationError : Response status code does not indicate success: 500 (Internal Server Error).
at <ScriptBlock>, <No file>: line 91
at AwaitState, <No file>: line 220
at OpenSession, <No file>: line 198
at Initialize, <No file>: line 59
at ClientContext, <No file>: line 26
at New-ClientContext, <No file>: line 27
at <ScriptBlock>, <No file>: line 10

the used yaml please provide the yaml that you used. It helps you put the yaml like this:

- task: ALOpsAppTest@1
  displayName: 'Run TestSuite ${{TestSuite}}'
  inputs:
    usedocker: true
    import_testtoolkit: false
    import_action: 'Skip'
    testpage: '130455'
    testsuite: 'DEFAULT'
    failed_test_action: 'Error'
    resultfilename: 'DEFAULT-TestResults.xml'
  continueOnError: false

the output (2024-01-23) Also the complete output is necessary for us to see what is going on. Also use backtics:

##[section]Starting: Run TestSuite
==============================================================================
Task         : ALOps App Test
Description  : Run AL-TestSuite for Business Central
Version      : 1.461.5118
Author       : Hodor
Help         : Run Business Central Test-Suite and collect results.
==============================================================================
*** Validate configuration
*** Task Inputs:

name                           value
----                           -----
usedocker                       True
fixed_tag                           
nav_serverinstance                  
tenant                       default
companyname                         
profile                             
bc_username                         
bc_password                         
bc_authentication    NavUserPassword
bc_webclient_url                    
testfilter                          
disabledtests                       
show_available_tests            True
import_testtoolkit             False
import_action                   Skip
testpage                      130455
testsuite                    DEFAULT
extensionid                         
installaltesttool               True
failed_test_action             Error
override_finsql_path                
resultfilename       TestResults.xml
testrunnercodeunitid                

*** For documentation, please visit   : https://www.alops.be/documentation

*** ALOps License:
  * Licensed To: prisma informatik GmbH (Organisation License)

*** Importing required PS-Functions
*** Working folder: C:\agent\_work\883\s
***
*** Prepare Docker Session
*** Connect Docker Session
*** Initiate Docker Session
*** Set Docker Container ErrorActionPreference = Stop
*** Copy Modules to Docker
*** Setup Docker Session
*** Checking for container-specific Test Assemblies
*** Using container-specific Test Assemblies
*** Import Modules in Docker Session
*** Import UI Client [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
*** Selected module: [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
*** Loading module: [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
*** Import NewtonSoft JSON [C:\Program Files\Microsoft Dynamics NAV\170\Service\Newtonsoft.Json.dll]
*** Import Modules in Docker Session: class_ClientContext
*** Test Context Url: https://74c231f5b549/BC/cs?tenant=default
*** Setup BC Credentials in Docker
*** Resolve Docker Credentials
*** Start AL-TestTool installation
*** Import NAV/BC Management DLL's [False]
*** ServiceTier not specified, loading default assemblies.
*** Load module [Microsoft.Dynamics.Nav.Management]
*** Selected module: [C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Management.dll]
*** Loading module: [C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Management.dll]
*** Load module [Microsoft.Dynamics.Nav.Apps.Management]
*** Selected module: [C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Apps.Management.dll]
*** Loading module: [C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Apps.Management.dll]
*** Check MockAssemblies.
*** Application requested for installation:
  * Any
  * Library Assert
  * Library Variable Storage
  * System Application Test
  * System Application Test Library
  * Test Runner
  * Tests-TestLibraries

*** Get Available Applications from [C:\Applications*\].
*** Get [*.app] files recursive from [C:\Applications*\]
*** Found [89] files.

[...]

*** Remove duplicate Apps
*** Substitute W1 extensions with localised versions if available
*** Available Applications:

[...]

*** Installing Applications:

Publisher Name                            Path                                                                         
--------- ----                            ----                                                                         
Microsoft Library Assert                  C:\Applications\TestFramework\TestLibraries\Assert\Microsoft_Library Asser...
Microsoft Library Variable Storage        C:\Applications\TestFramework\TestLibraries\Variable Storage\Microsoft_Lib...
Microsoft Any                             C:\Applications\TestFramework\TestLibraries\Any\Microsoft_Any.app            
Microsoft System Application Test Library C:\Applications\System Application\Test\Microsoft_System Application Test ...
Microsoft Tests-TestLibraries             C:\Applications\BaseApp\Test\Microsoft_Tests-TestLibraries.app               
Microsoft Test Runner                     C:\Applications\TestFramework\TestRunner\Microsoft_Test Runner.app           
Microsoft System Application Test         C:\Applications\System Application\Test\Microsoft_System Application Test.app

*** Get App Info: C:\Applications\TestFramework\TestLibraries\Assert\Microsoft_Library Assert.app

*** Get App Tenant Info
*** Sync App: [Microsoft / Library Assert]
WARNUNG: Cannot synchronize the extension Library Assert because it is already synchronized.
***
*** Get App Info: C:\Applications\TestFramework\TestLibraries\Variable Storage\Microsoft_Library Variable Storage.app

[...]

*** Get App Tenant Info
*** Publish App: [Microsoft / System Application Test]
*** Sync App: [Microsoft / System Application Test]
*** Install App: [Microsoft / System Application Test]
***

*** AL-TestTool installation finished.
*** Set TCP Keep-Alive
*** Start TestSuite
*** Running TestSuite with authentication [NavUserPassword] with User [admin]
*** Run warm-up with page [130455]
*** Warm-up Client Context
*** Create Client Context
*** Get Available Tests for suite [DEFAULT] with page [130455]
Show form {0001fd97-0000-0001-0c00-0000836bd2d2}
Title: Command Line Test Tool Edit - Command Line Test Tool - DEFAULT
*** Set [CurrentSuiteName] => [DEFAULT]
[
    {
        "Tests":  [
                      "ReportIntegrationTest",
                      "ReportIntegrationDisabledTest",
                      "ReportIntegrationWithDocAttachmentTest"
                  ],
        "Name":  "ReportIntegrationTest",
        "Id":  "51501"
    },
    {
        "Tests":  [
                      "CheckSalesDocumentHeaders",
                      "CheckSalesDocumentLines",
                      "CheckPurchDocumentHeaders",
                      "CheckPurchDocumentLines",
                      "CheckContactTables"
                  ],
        "Name":  "CheckTransferfields",
        "Id":  "51502"
    },
    {
        "Tests":  [
                      "CreateUpdateEntryOnApplyOfPurchInvoiceTest",
                      "CreatePurchLineWithVatGroupMappingTest"
                  ],
        "Name":  "Inc. Doc. Proc. Test",
        "Id":  "51503"
    },
    {
        "Tests":  [
                      "FileCabinetFilteringTest"
                  ],
        "Name":  "Extensibility Tests",
        "Id":  "51504"
    }
]
*** Run Available Tests for suite [DEFAULT] with page [130455]
*** Connect TestClient with page [130455]
Show form {0001fd97-0000-0001-0c00-0000836bd2d2}
Title: Command Line Test Tool Edit - Command Line Test Tool - DEFAULT
  Codeunit 51501 ReportIntegrationTest Success (28.83 seconds)
  Codeunit 51502 CheckTransferfields Success (2.212 seconds)
  Codeunit 51503 Inc. Doc. Proc. Test Success (14.676 seconds)
  Codeunit 51504 Extensibility Tests Success (1.766 seconds)
True
*** XUnit Test results: c:\Run\DevOps\TestResults.xml
<?xml version="1.0" encoding="UTF-8"?>
<assemblies>
  <assembly name="51501 ReportIntegrationTest" test-framework="PS Test Runner" run-date="2024-01-23" run-time="08:53:13" total="3" passed="3" failed="0" skipped="0" time="27.074">
    <collection name="ReportIntegrationTest" total="3" passed="3" failed="0" skipped="0" time="27.074" Skipped="0">
      <test name="ReportIntegrationTest:ReportIntegrationTest" method="ReportIntegrationTest" time="25.204" result="Pass" />
      <test name="ReportIntegrationTest:ReportIntegrationDisabledTest" method="ReportIntegrationDisabledTest" time="0.387" result="Pass" />
      <test name="ReportIntegrationTest:ReportIntegrationWithDocAttachmentTest" method="ReportIntegrationWithDocAttachmentTest" time="1.483" result="Pass" />
    </collection>
  </assembly>
  <assembly name="51502 CheckTransferfields" test-framework="PS Test Runner" run-date="2024-01-23" run-time="08:53:42" total="5" passed="5" failed="0" skipped="0" time="0.706">
    <collection name="CheckTransferfields" total="5" passed="5" failed="0" skipped="0" time="0.706" Skipped="0">
      <test name="CheckTransferfields:CheckSalesDocumentHeaders" method="CheckSalesDocumentHeaders" time="0.126" result="Pass" />
      <test name="CheckTransferfields:CheckSalesDocumentLines" method="CheckSalesDocumentLines" time="0.187" result="Pass" />
      <test name="CheckTransferfields:CheckPurchDocumentHeaders" method="CheckPurchDocumentHeaders" time="0.173" result="Pass" />
      <test name="CheckTransferfields:CheckPurchDocumentLines" method="CheckPurchDocumentLines" time="0.187" result="Pass" />
      <test name="CheckTransferfields:CheckContactTables" method="CheckContactTables" time="0.033" result="Pass" />
    </collection>
  </assembly>
  <assembly name="51503 Inc. Doc. Proc. Test" test-framework="PS Test Runner" run-date="2024-01-23" run-time="08:53:44" total="2" passed="2" failed="0" skipped="0" time="13.043">
    <collection name="Inc. Doc. Proc. Test" total="2" passed="2" failed="0" skipped="0" time="13.043" Skipped="0">
      <test name="Inc. Doc. Proc. Test:CreateUpdateEntryOnApplyOfPurchInvoiceTest" method="CreateUpdateEntryOnApplyOfPurchInvoiceTest" time="12.883" result="Pass" />
      <test name="Inc. Doc. Proc. Test:CreatePurchLineWithVatGroupMappingTest" method="CreatePurchLineWithVatGroupMappingTest" time="0.16" result="Pass" />
    </collection>
  </assembly>
  <assembly name="51504 Extensibility Tests" test-framework="PS Test Runner" run-date="2024-01-23" run-time="08:53:59" total="1" passed="1" failed="0" skipped="0" time="0.17">
    <collection name="Extensibility Tests" total="1" passed="1" failed="0" skipped="0" time="0.17" Skipped="0">
      <test name="Extensibility Tests:FileCabinetFilteringTest" method="FileCabinetFilteringTest" time="0.17" result="Pass" />
    </collection>
  </assembly>
</assemblies>

*** Get XUnit Test file from [c:\Run\DevOps\TestResults.xml] to [C:\agent\_work\883\s\TestResults.xml]
ComputerType           : Container
ComputerName           : 74c231f5b549291cd05c232d9b93aa459aa7202d72488c1e69e16ab79af4e8ed
ContainerId            : 74c231f5b549291cd05c232d9b93aa459aa7202d72488c1e69e16ab79af4e8ed
VMName                 : 
VMId                   : 
ConfigurationName      : 
InstanceId             : 3f119c5d-87c4-4f22-b831-7e155e10f2fa
Id                     : 1
Name                   : WinRM1
Availability           : Available
ApplicationPrivateData : {DebugMode, DebugStop, UnhandledBreakpointMode, PSVersionTable...}
Runspace               : System.Management.Automation.RemoteRunspace
State                  : Opened
IdleTimeout            : -1
OutputBufferingMode    : 
DisconnectedOn         : 
ExpiresOn              : 

*** Disconnect Docker session.
*** Docker session removed.
*** Running Tests Completed.
*** Cleanup VSTS Environment: True
##[section]Finishing: Run TestSuite

the output (2024-07-10) Also the complete output is necessary for us to see what is going on. Also use backtics:

##[section]Starting: Run TestSuite
==============================================================================
Task         : ALOps App Test
Description  : Run AL-TestSuite for Business Central
Version      : 1.465.6167
Author       : Hodor
Help         : Run Business Central Test-Suite and collect results.
==============================================================================
*** Validate configuration
*** Task Inputs:

name                           value
----                           -----
usedocker                       True
fixed_tag                           
nav_serverinstance                  
tenant                       default
companyname                         
profile                             
bc_username                         
bc_password                         
bc_authentication    NavUserPassword
bc_webclient_url                    
testfilter                          
disabledtests                       
show_available_tests            True
import_testtoolkit             False
import_action                   Skip
testpage                      130455
testsuite                    DEFAULT
extensionid                         
installaltesttool               True
failed_test_action             Error
override_finsql_path                
resultfilename       TestResults.xml
testrunnercodeunitid                
allow_zero_tests               False

True
*** For documentation, please visit   : https://www.alops.be/documentation

*** Importing required PS-Functions
*** Working folder: C:\agent\_work\2716\s
***
*** Prepare Docker Session
*** Connect Docker Session
*** Initiate Docker Session
*** Set Docker Container ErrorActionPreference = Stop
*** Container Session Powershell Version

Name  : PSVersion
Value : 5.1.17763.5933

Name  : PSEdition
Value : Desktop

Name  : PSCompatibleVersions
Value : {1.0, 2.0, 3.0, 4.0...}

Name  : BuildVersion
Value : 10.0.17763.5933

Name  : CLRVersion
Value : 4.0.30319.42000

Name  : WSManStackVersion
Value : 3.0

Name  : PSRemotingProtocolVersion
Value : 2.3

Name  : SerializationVersion
Value : 1.1.0.1

*** Copy Modules to Docker
*** Setup Docker Session
*** Checking for container-specific Test Assemblies
*** Using container-specific Test Assemblies
*** Import Modules in Docker Session
*** Import UI Client [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
*** Selected module: [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
*** Loading module: [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
*** Import NewtonSoft JSON [C:\Program Files\Microsoft Dynamics NAV\170\Service\Newtonsoft.Json.dll]
*** Import Modules in Docker Session: class_ClientContext
*** Test Context Url: https://7d415a8890d6/BC/cs?tenant=default
*** Setup BC Credentials in Docker
*** Resolve Docker Credentials
*** Start AL-TestTool installation
*** Import NAV/BC Management DLL's [False]
*** ServiceTier not specified, loading default assemblies.
*** Load module [Microsoft.Dynamics.Nav.Management]
*** BC Management DLL: C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Management.dll
*** Selected module: [C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Management.dll]
*** Loading module: [C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Management.dll]
*** Load module [Microsoft.Dynamics.Nav.Apps.Management]
*** BC App Management DLL: C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Apps.Management.dll
*** Selected module: [C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Apps.Management.dll]
*** Loading module: [C:\Program Files\Microsoft Dynamics NAV\170\Service\Microsoft.Dynamics.Nav.Apps.Management.dll]
*** Check MockAssemblies.
*** Application requested for installation:
  * Any
  * Library Assert
  * Library Variable Storage
  * System Application Test
  * System Application Test Library
  * Test Runner
  * Tests-TestLibraries

*** Get Available Applications from [C:\Applications*\].
*** Get [*.app] files recursive from [C:\Applications*\]
*** Found [89] files.

[...]

*** Remove duplicate Apps
*** Substitute W1 extensions with localised versions if available
*** Available Applications:

[...]

*** Installing Applications:

ProcessOrder : -81
Publisher    : Microsoft
Name         : Library Assert
Path         : C:\Applications\TestFramework\TestLibraries\Assert\Microsoft_Library Assert.app

ProcessOrder : -78
Publisher    : Microsoft
Name         : Library Variable Storage
Path         : C:\Applications\TestFramework\TestLibraries\Variable Storage\Microsoft_Library Variable Storage.app

ProcessOrder : -71
Publisher    : Microsoft
Name         : Any
Path         : C:\Applications\TestFramework\TestLibraries\Any\Microsoft_Any.app

ProcessOrder : -69
Publisher    : Microsoft
Name         : System Application Test Library
Path         : C:\Applications\System Application\Test\Microsoft_System Application Test Library.app

ProcessOrder : -40
Publisher    : Microsoft
Name         : Tests-TestLibraries
Path         : C:\Applications\BaseApp\Test\Microsoft_Tests-TestLibraries.app

ProcessOrder : -2
Publisher    : Microsoft
Name         : Test Runner
Path         : C:\Applications\TestFramework\TestRunner\Microsoft_Test Runner.app

ProcessOrder : -1
Publisher    : Microsoft
Name         : System Application Test
Path         : C:\Applications\System Application\Test\Microsoft_System Application Test.app

*** Get App Info: C:\Applications\TestFramework\TestLibraries\Assert\Microsoft_Library Assert.app

*** Get App Tenant Info
*** Sync App: [Microsoft / Library Assert]
WARNUNG: Cannot synchronize the extension Library Assert because it is already synchronized.
***
*** Get App Info: C:\Applications\TestFramework\TestLibraries\Variable Storage\Microsoft_Library Variable Storage.app
*** Get App Tenant Info
*** Sync App: [Microsoft / Library Variable Storage]
WARNUNG: Cannot synchronize the extension Library Variable Storage because it is already synchronized.
***
*** Get App Info: C:\Applications\TestFramework\TestLibraries\Any\Microsoft_Any.app
*** Get App Tenant Info
*** Sync App: [Microsoft / Any]
WARNUNG: Cannot synchronize the extension Any because it is already synchronized.
***
*** Get App Info: C:\Applications\System Application\Test\Microsoft_System Application Test Library.app
*** Get App Tenant Info
*** Sync App: [Microsoft / System Application Test Library]
WARNUNG: Cannot synchronize the extension System Application Test Library because it is already synchronized.
***
*** Get App Info: C:\Applications\BaseApp\Test\Microsoft_Tests-TestLibraries.app
*** Get App Tenant Info
*** Sync App: [Microsoft / Tests-TestLibraries]
WARNUNG: Cannot synchronize the extension Tests-TestLibraries because it is already synchronized.
***
*** Get App Info: C:\Applications\TestFramework\TestRunner\Microsoft_Test Runner.app
*** Get App Tenant Info
*** Sync App: [Microsoft / Test Runner]
WARNUNG: Cannot synchronize the extension Test Runner because it is already synchronized.
***
*** Get App Info: C:\Applications\System Application\Test\Microsoft_System Application Test.app
*** Get App Tenant Info
*** Publish App: [Microsoft / System Application Test]
*** Sync App: [Microsoft / System Application Test]
*** Install App: [Microsoft / System Application Test]
***

*** AL-TestTool installation finished.
*** Set TCP Keep-Alive
*** Start TestSuite
*** Running TestSuite with authentication [NavUserPassword] with User [admin]
*** Run warm-up with page [130455]
*** Warm-up Client Context
CommunicationError : Response status code does not indicate success: 500 (Internal Server Error).
at <ScriptBlock>, <No file>: line 91
at AwaitState, <No file>: line 220
at OpenSession, <No file>: line 198
at Initialize, <No file>: line 59
at ClientContext, <No file>: line 26
at New-ClientContext, <No file>: line 27
at <ScriptBlock>, <No file>: line 10
*** Warm-Up Exception: ClientSession State is Uninitialized (Wait time 10 seconds)
*** Warm-up Client Context failed: retry
*** Warm-up Client Context
CommunicationError : Response status code does not indicate success: 500 (Internal Server Error).
at <ScriptBlock>, <No file>: line 91
at AwaitState, <No file>: line 220
at OpenSession, <No file>: line 198
at Initialize, <No file>: line 59
at ClientContext, <No file>: line 26
at New-ClientContext, <No file>: line 27
at <ScriptBlock>, <No file>: line 10
*** Warm-Up Exception: ClientSession State is Uninitialized (Wait time 10 seconds)
*** Warm-up Client Context failed: retry
*** Warm-up Client Context
CommunicationError : Response status code does not indicate success: 500 (Internal Server Error).
at <ScriptBlock>, <No file>: line 91
at AwaitState, <No file>: line 220
at OpenSession, <No file>: line 198
at Initialize, <No file>: line 59
at ClientContext, <No file>: line 26
at New-ClientContext, <No file>: line 27
at <ScriptBlock>, <No file>: line 10
*** Warm-Up Exception: ClientSession State is Uninitialized (Wait time 10 seconds)
*** Warm-up Client Context failed: retry
*** Warm-up Client Context
CommunicationError : Response status code does not indicate success: 500 (Internal Server Error).
at <ScriptBlock>, <No file>: line 91
at AwaitState, <No file>: line 220
at OpenSession, <No file>: line 198
at Initialize, <No file>: line 59
at ClientContext, <No file>: line 26
at New-ClientContext, <No file>: line 27
at <ScriptBlock>, <No file>: line 10
*** Warm-Up Exception: ClientSession State is Uninitialized (Wait time 10 seconds)
*** Warm-up Client Context failed: retry
*** Warm-up Client Context
CommunicationError : Response status code does not indicate success: 500 (Internal Server Error).
at <ScriptBlock>, <No file>: line 91
at AwaitState, <No file>: line 220
at OpenSession, <No file>: line 198
at Initialize, <No file>: line 59
at ClientContext, <No file>: line 26
at New-ClientContext, <No file>: line 27
at <ScriptBlock>, <No file>: line 10
*** Warm-Up Exception: ClientSession State is Uninitialized (Wait time 10 seconds)
*** Warm-up Client Context failed: retry
##[error]Warm-up Client Context failed
##[error]Da der Sitzungszustand für die Sitzung WinRM1, 6b9598e2-aa8d-4343-9e08-b606b4bda667, 7d415a8890d6138f3604cfa4c8b0a8f21115f4732567f239b2f383afb4855b5c nicht "Open" ist, können Sie keinen Befehl in der Sitzung ausführen. Sitzungszustand: Closing.
##[section]Finishing: Run TestSuite

[waldo1001]

Can you investigate the eventlogs? just make sure to save eventlog on the remove-docker step. You can set it "OnFailure" or "Always".

[waldo1001]

@DanielGoehler , did you have a chance to look at this?

[DanielGoehler]

@waldo1001 I check again, now the tests are working again.