ALOpsAppTest not working on Next Major (BC v20)

[Arthurvdv]

Describe the bug The ALOpsAppTest doesn't seems to work on the Next Major version of Business Central (v20). It doesn't throw an error, but execute zero tests. The Next Minor and Current works without a problem and all tests are executed.

the used yaml please provide the yaml that you used. It helps you put the yaml like this:

steps:
  - task: ALOpsAppPublish@1
    displayName: 'Install AL TestTool'
    inputs:
      usedocker: true
      installaltesttool: true
      skip_verification: true
      install_al_app_names: |
        Tests-TestLibraries
        System Application Test
        System Application Test Library
        Any
        Library Assert
        Test Runner
        Library Variable Storage
        Permissions Mock

  - task: ALOpsAppTest@1
    displayName: 'Run Test Suite'
    inputs:
      usedocker: true
      import_action: "Skip"
      import_testtoolkit: false
      testpage: '130455'
      testsuite: 'DEFAULT'
      failed_test_action: 'Error'
    continueOnError: false

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'
    inputs:
      testResultsFormat: XUnit
      testResultsFiles: '**/TestResults.xml'
      failTaskOnFailedTests: true
      testRunTitle: 'BC Test Results: $(Build.BuildId)'

codeunit 80000 "TST Test Install"
{
    Subtype = Install;

    trigger OnInstallAppPerCompany()
    begin
        SetupTestSuite();
    end;

    local procedure SetupTestSuite()
    var
        ALTestSuite: Record "AL Test Suite";
        TestSuiteMgt: Codeunit "Test Suite Mgt.";
        SuiteName: Code[10];
    begin
        SuiteName := 'DEFAULT';

        if ALTestSuite.Get(SuiteName) then
            ALTestSuite.Delete(true);

        TestSuiteMgt.CreateTestSuite(SuiteName);
        ALTestSuite.Get(SuiteName);

        TestSuiteMgt.SelectTestMethodsByRange(ALTestSuite, '80000..89999');
    end;
}

the output Also the complete output is necessary for us to see what is going on. Also use backtics:

2022-01-07T09:29:03.8166709Z ##[section]Starting: Run Test Suite
2022-01-07T09:29:03.8300533Z ==============================================================================
2022-01-07T09:29:03.8300740Z Task         : ALOps App Test
2022-01-07T09:29:03.8300914Z Description  : Run AL-TestSuite for Business Central
2022-01-07T09:29:03.8301070Z Version      : 1.446.2728
2022-01-07T09:29:03.8301202Z Author       : Hodor
2022-01-07T09:29:03.8301378Z Help         : Run Business Central Test-Suite and collect results.
2022-01-07T09:29:03.8301586Z ==============================================================================
2022-01-07T09:29:05.6152063Z *** Validate configuration
2022-01-07T09:29:05.8305225Z *** Task Inputs:
2022-01-07T09:29:05.8342271Z 
2022-01-07T09:29:05.8425671Z name                           value
2022-01-07T09:29:05.8428036Z ----                           -----
2022-01-07T09:29:05.8428859Z usedocker                       True
2022-01-07T09:29:05.8430472Z fixed_tag                           
2022-01-07T09:29:05.8432629Z nav_serverinstance             BC140
2022-01-07T09:29:05.8433007Z tenant                       default
2022-01-07T09:29:05.8433939Z companyname                         
2022-01-07T09:29:05.8434969Z profile                             
2022-01-07T09:29:05.8436136Z bc_username                         
2022-01-07T09:29:05.8437131Z bc_password                         
2022-01-07T09:29:05.8438175Z bc_authentication    NavUserPassword
2022-01-07T09:29:05.8439240Z bc_webclient_url                    
2022-01-07T09:29:05.8440316Z testfilter                          
2022-01-07T09:29:05.8442072Z disabledtests                       
2022-01-07T09:29:05.8443673Z show_available_tests            True
2022-01-07T09:29:05.8445789Z import_testtoolkit             False
2022-01-07T09:29:05.8447702Z import_action                   Skip
2022-01-07T09:29:05.8449653Z testpage                      130455
2022-01-07T09:29:05.8450852Z testsuite                    DEFAULT
2022-01-07T09:29:05.8452579Z extensionid                         
2022-01-07T09:29:05.8453160Z installaltesttool              False
2022-01-07T09:29:05.8454119Z failed_test_action             Error
2022-01-07T09:29:05.8455104Z override_finsql_path                
2022-01-07T09:29:05.8456104Z resultfilename       TestResults.xml
2022-01-07T09:29:05.8456250Z 
2022-01-07T09:29:05.8470073Z 
2022-01-07T09:29:05.8475180Z 
2022-01-07T09:29:05.8580461Z *** For documentation, please visit   : https://www.alops.be/documentation
2022-01-07T09:29:05.8582281Z 
2022-01-07T09:29:06.2726482Z *** ALOps License:
2022-01-07T09:29:06.2739628Z   * Licensed To: Van Roey (Collection License)
2022-01-07T09:29:06.2742982Z 
2022-01-07T09:29:06.2814399Z *** Importing required PS-Functions
2022-01-07T09:29:06.4636607Z *** Working folder: C:\azure-vsts-agent-build\_work\93\s
2022-01-07T09:29:06.4641906Z ***
2022-01-07T09:29:06.4647437Z *** Prepare Docker Session
2022-01-07T09:29:06.6509868Z *** Connect Docker Session
2022-01-07T09:29:06.6561688Z *** Initiate Docker Session
2022-01-07T09:29:06.8832811Z *** Set Docker Container ErrorActionPreference = Stop
2022-01-07T09:29:07.1871709Z *** Copy Modules to Docker
2022-01-07T09:29:09.0010256Z *** Setup Docker Session
2022-01-07T09:29:09.0237259Z *** Checking for container-specific Test Assemblies
2022-01-07T09:29:09.0246389Z *** Using container-specific Test Assemblies
2022-01-07T09:29:09.0549552Z *** Import Modules in Docker Session
2022-01-07T09:29:09.1040421Z *** Import UI Client [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
2022-01-07T09:29:09.1051139Z VERBOSE: Loading module from path 'C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll'.
2022-01-07T09:29:09.2700150Z *** Import NewtonSoft JSON [C:\Program Files\Microsoft Dynamics NAV\200\Service\Newtonsoft.Json.dll]
2022-01-07T09:29:09.2803328Z *** Import Modules in Docker Session: class_ClientContext
2022-01-07T09:29:11.1659652Z *** Test Context Url: https://5966856a16be/BC/cs?tenant=default
2022-01-07T09:29:11.1664350Z *** Setup BC Credentials in Docker
2022-01-07T09:29:11.3979751Z *** Resolve Docker Credentials
2022-01-07T09:29:11.4025112Z *** Set TCP Keep-Alive
2022-01-07T09:29:11.4124616Z *** Start TestSuite
2022-01-07T09:29:11.4705727Z *** Running TestSuite with authentication [NavUserPassword] with User [admin]
2022-01-07T09:29:11.4713205Z *** Run warm-up with page [130455]
2022-01-07T09:29:11.4722641Z *** Warm-up Client Context
2022-01-07T09:29:22.9967752Z *** Create Client Context
2022-01-07T09:29:24.1041758Z *** Get Available Tests for suite [DEFAULT] with page [130455]
2022-01-07T09:29:24.5527492Z Show form 0001fd97-0000-0001-0c00-0000836bd2d2
2022-01-07T09:29:24.6121743Z Title: Command Line Test Tool Edit - Command Line Test Tool - DEFAULT
2022-01-07T09:29:24.7093941Z *** Set [CurrentSuiteName] => [DEFAULT]
2022-01-07T09:29:26.1081813Z *** Run Available Tests for suite [DEFAULT] with page [130455]
2022-01-07T09:29:26.1498543Z *** Connect TestClient with page [130455]
2022-01-07T09:29:26.2544184Z Show form 0001fd97-0000-0001-0c00-0000836bd2d2
2022-01-07T09:29:26.2586078Z Title: Command Line Test Tool Edit - Command Line Test Tool - DEFAULT
2022-01-07T09:29:27.4758778Z True
2022-01-07T09:29:27.4812390Z *** XUnit Test results: c:\Run\DevOps\TestResults.xml
2022-01-07T09:29:27.4872878Z <?xml version="1.0" encoding="UTF-8"?>
2022-01-07T09:29:27.4873195Z <assemblies />
2022-01-07T09:29:27.4873287Z 
2022-01-07T09:29:27.4917652Z *** Get XUnit Test file from [c:\Run\DevOps\TestResults.xml] to [C:\azure-vsts-agent-build\_work\93\s\TestResults.xml]
2022-01-07T09:29:27.6448656Z 
2022-01-07T09:29:27.6453418Z *** Disconnect Docker session.
2022-01-07T09:29:27.6468342Z *** Docker session removed.
2022-01-07T09:29:27.6535471Z *** Running Tests Completed.
2022-01-07T09:29:27.6588372Z *** Cleanup VSTS Environment: True
2022-01-07T09:29:29.9370937Z ##[section]Finishing: Run Test Suite

Expected behavior Running the tests and/or reporting back it has not run alle the tests successfully.

Maybe a feature request is hidden inside this bug; The pipeline doesn't thrown an error on executing zero tests, what could give a false sense of safety. The status is green, but without validating any tests is not an ideal situation? For me it should thrown an error when zero tests are returned (why execute ALOpsAppTest with zero tests?). If there's a use case for this, maybe an a extra parameter: DoNotErrorOnExecutingZeroTests with a default als false?

[waldo1001]

Hey,

we are not able to repro this problem. would you be able to send me a repro?

[waldo1001]

@Arthurvdv ?

[waldo1001]

I have closed the issue - please reopen if you're able to set up a repro..

[Arthurvdv]

@waldo1001: This was a false positive, sorry the inconvenience and trouble!

(i thought the Next Major branch was up-to-date and the same as the Main branch. After more detailed inspection someone had experimented with the "Test Install" codeunit in the Next Major branch, what broke the install of the Test Suite).