Question: what is the best/efficient way to populated a test suite with all tests present?

[lvanvugt]

I would like to efficiently populate the test suite DEFAULT with all tests that reside in the database. My specific case is that I want to run all standard tests (might not be a surprise). I did set up my ALOpsAppTest steps as follows

  - task: ALOpsAppTest@1
    displayName: 'ALOPS - Run TestSuite'
    inputs:
      usedocker: true
      import_testtoolkit: false
      import_action: 'Skip'
      testpage: '130455'
      failed_test_action: 'Ignore'

But this seems not to populate the test suite. Is the only way to get this done by means of an install codeunit?

[waldo1001]

Luc,

I'll set up an example and share it with you! There are a few settings that are not that clear, because of backwards compatibility with BCv14 and such ... I get that now. At this moment, best example is this one: https://github.com/HodorNV/ALOps/blob/master/Examples/06_UseDocker_TestTool_v15_Multiroot.yml

[waldo1001]

I have a working example - but it's not running any tests 🤣.

Reason is because MS doesn't include an install-codeunit that adds the tests to a test-suite .. .

Can't find any solution in BcContainerHelper either.

We'll see what we can do!

[gntpet]

we run out tests like this. but u must have adjusted testrunner within your code base (i think @waldo1001 you gave us this fob)

- task: ALOpsAppTest@1
  displayName: 'Run TestSuite'
  inputs:
    usedocker: true
    testfilter: '(50000..70099|70301..99999)&<>53051'
    import_testtoolkit: false
    show_available_tests: true
    failed_test_action: Error 

[waldo1001]

Thanks @gintuoklis ,

but isn't this v14?

[gntpet]

Yes, it is v14.

[waldo1001]

Well, since v15, the way for running tests is turned upside down. And at this moment, I don't see a default way to add tests other than an install app ..

yet ...

[lvanvugt]

That's what I was doing. Posing the question hoping for a more efficient (and less time costly) way.

I am setting up my TextFixtureInitializer project to be able to run all MS tests. The install codeunit can be seen here https://github.com/fluxxus-nl/TestFixtureInitializer/blob/master/TestFixtureInitializer/Test/src/codeunit/TestInstall.Codeunit.al. And all pipelines (for each MS test app one) here https://github.com/fluxxus-nl/TestFixtureInitializer/tree/master/TestFixtureInitializer/Azure%20Pipelines.

BTW: here you can find the ADO project (dashboards): https://dev.azure.com/fluxxus-nl/TestFixtureInitializer/_dashboards/directory

[lvanvugt]

we run out tests like this. but u must have adjusted testrunner within your code base (i think @waldo1001 you gave us this fob)

Indeed on BC14 this is so much easier.

[lvanvugt]

part of an upcoming release

[did you notice the typo in upcomming?]

Great and thanx for the effort!

But for your information, and you might have seen this on BC yammer @waldo1001, it has become clear that running all MS test in one run will not give good results. In Nikola's words:

Internally in Microsoft we use buckets from above to run tests. We publish all apps and then execute the codeunits cross apps using buckets from above. This is done because of legacy reasons (before test extensions were implemented).

Unfortunately there is a large number of tests that have a cross test dependency or are doing "destructive" things to the environment, so if you want to run all tests you should follow the way we do it, otherwise you will hit some instabilities.

Consequence is that it will not be useful to run all the MS tests in one stretch at this point in time. That's what the various pipelines runs on my TestFixtureInitializer project also show, unfortunately.

Next week I am having a meeting with Nikola and Freddy on this and some other test automation matters.

[lvanvugt]

Meanwhile I tried an idea which might be a good workaround, but ALOps seems not to support it. Therefore I have created a new issue. See #315.

[lvanvugt]

@waldo1001, could you elaborate on what the addition to the latest release entails and how to use it? Thanx in advance.

[waldo1001]

Do know we're revising the testability as we speak as per your request to run per ExtensionId .. (including concatenating the results)..

But this is what we have now:

  - task: ALOpsAppTest@1
    displayName: 'ALOPS - Run TestSuite'
    inputs:
      usedocker: true
      import_action: "Skip"
      import_testtoolkit: false
      testpage: '130455'
      testsuite: 'DEFAULT'
      testfilter: '<>0'
      failed_test_action: 'Ignore'
      show_available_tests: false
    continueOnError: true
    timeoutInMinutes: 360

By stating the testfilter, it will basically do a "get all test codeunits" with that filter applied.

If you remove the parameter, it won't add anything.

[waldo1001]

@lvanvugt ,

we noticed there is still an issue with loading our app to be able to do the above. Looking at it as we speak ..

[waldo1001]

@lvanvugt ,

when thinking about this ... Since we're hacking our way through this, and with the suggestion from Freddy in mind - would it be enough to make a way to add/run tests per extensionid, and abandon this approach to add tests to a testsuite?

[lvanvugt]

Yep, fine with me.

[lvanvugt]

One extra note: I would still want to be able to set a filter so I can run a subset of the tests.

[waldo1001]

If the page allows it to combine extensionid and filter, we'll do .. but I'm not sure it will allow it..

[lvanvugt]

@waldo1001 today I did update my install codeunit and the pipeline .yml to the following. As a result the DEFAULT test suite was populated with all tests in the various (i.e. for the exmple below 2) test apps present. So, this fullfills the need as described in the initial issue. Of course having the possibility to reference extension id's is something that still is very welcome to provide. We might close this issue and create a new one for that.

Install codeunit

codeunit 97050 "FLX Tfi Test Install"
{
    Subtype = Install;

    trigger OnInstallAppPerCompany()
    begin
        IsFirstTime := true;

        SetupTestSuites();
    end;

    local procedure SetupTestSuites()
    begin
        AddToTestSuite('DEFAULT', '1fb2e583-f2c3-4484-a8d4-387e3adce46d'); //  Tests-Bank
        AddToTestSuite('DEFAULT', '69837cc7-28e9-4db0-9a0f-6616f259c91a'); //  Tests-Cash Flow
        AddToTestSuite('DEFAULT', '16419195-5f01-494f-9b02-34363dade478'); //  Tests-Cost Accounting
        AddToTestSuite('DEFAULT', '155bc500-e420-4113-803e-7aa8e8eea112'); //  Tests-CRM integration
        AddToTestSuite('DEFAULT', '85125fdc-eb14-4f16-8c5d-6e4589aceb1d'); //  Tests-Data Exchange
        AddToTestSuite('DEFAULT', '6d9b6d22-97e0-4774-982a-2097fd660f97'); //  Tests-Dimension
        AddToTestSuite('DEFAULT', 'fa3e2564-a39e-417f-9be6-c0dbe3d94069'); //  Tests-ERM
        AddToTestSuite('DEFAULT', 'c81764a5-be79-4d50-ba3e-4ade02073780'); //  Tests-Fixed Asset
        AddToTestSuite('DEFAULT', '790cf8ce-f921-42a8-8c34-a0af2ddf6265'); //  Tests-General Journal
        AddToTestSuite('DEFAULT', '474a0e0a-d177-4c70-aae0-4ff674a04e74'); //  Tests-Graph
        AddToTestSuite('DEFAULT', '2b634ad5-63df-4a3c-9e35-e347f646e940'); //  Tests-Integration
        AddToTestSuite('DEFAULT', 'cc2187fe-1b59-4f29-8a75-76d76c88c6dc'); //  Tests-Invoicing
        AddToTestSuite('DEFAULT', '9bf23d52-8194-4857-8896-5e48b24493f6'); //  Tests-Job
        AddToTestSuite('DEFAULT', 'bd6c6741-7734-4109-bf2d-db58545d482c'); //  Tests-Local
        AddToTestSuite('DEFAULT', '74e323c4-70a3-49ce-b18e-fe9ccaff01d3'); //  Tests-Marketing
        AddToTestSuite('DEFAULT', '4564dd2f-e8b2-41ff-9905-9d7a950475a5'); //  Tests-Misc
        AddToTestSuite('DEFAULT', 'c49d9d12-1c94-4362-8bf7-3cce15be54dc'); //  Tests-Monitor Sensitive Fields
        AddToTestSuite('DEFAULT', 'd94a0cee-f211-4941-b609-7c984f85ad03'); //  Tests-Permissions
        AddToTestSuite('DEFAULT', 'e94328f1-bf57-43d3-a862-93d54f83a9d7'); //  Tests-Physical Inventory
        AddToTestSuite('DEFAULT', 'fdddc5f1-3294-485f-aed1-0b798b6edd69'); //  Tests-Prepayment
        AddToTestSuite('DEFAULT', '7fe10be7-575f-4cce-957b-bcd0df8f9bd7'); //  Tests-Rapid Start
        AddToTestSuite('DEFAULT', '79447c3f-14b4-4c10-8779-03911f30ab26'); //  Tests-Report
        AddToTestSuite('DEFAULT', '0243a4ea-21b3-4fa8-ad70-9915a4400be0'); //  Tests-Resource
        AddToTestSuite('DEFAULT', 'ab333b30-f600-4d0e-a519-73684ff42504'); //  Tests-Reverse
        AddToTestSuite('DEFAULT', '6f0b528b-a4de-4ccd-a948-d1350e228e4a'); //  Tests-SCM
        AddToTestSuite('DEFAULT', 'c9bea3a6-a9af-499f-b78a-130a42eceedd'); //  Tests-SMB
        AddToTestSuite('DEFAULT', 'c49d9d39-1c94-46d2-8bf7-3cce15ba54dc'); //  Tests-SMTP
        AddToTestSuite('DEFAULT', '5d86850b-0d76-4eca-bd7b-951ad998e997'); //  Tests-TestLibraries
        AddToTestSuite('DEFAULT', 'd0e99b97-089b-449f-a0f5-a2ab994dbfd7'); //  Tests-Upgrade
        AddToTestSuite('DEFAULT', '9cc7e87b-8ba9-4c23-82d4-8af3e919c84b'); //  Tests-User
        AddToTestSuite('DEFAULT', '0f0955b8-92e2-4ce2-a580-3c4583dde9ae'); //  Tests-VAT
        AddToTestSuite('DEFAULT', 'c4795dd0-aee3-47cc-b020-2ee93a47d4c4'); //  Tests-Workflow
    end;

    local procedure AddToTestSuite(SuiteName: Text; ExtensionID: Text)
    var
        ALTestSuite: Record "AL Test Suite";
        TestSuiteMgt: Codeunit "Test Suite Mgt.";
        SuiteNameCode: Code[10];
        Module: ModuleInfo;
    begin
        if not NavApp.GetModuleInfo(ExtensionID, Module) then
            exit;

        SuiteNameCode := CopyStr(SuiteName, 1, MaxStrLen(SuiteNameCode));

        if IsFirstTime then begin
            if ALTestSuite.Get(SuiteNameCode) then
                ALTestSuite.Delete(true);
            TestSuiteMgt.CreateTestSuite(SuiteNameCode);
            IsFirstTime := false;
        end;

        ALTestSuite.Get(SuiteNameCode);
        TestSuiteMgt.SelectTestMethodsByExtension(ALTestSuite, ExtensionID);
    end;

    var
        IsFirstTime: Boolean;
}

Pipeline .yml

name: $(Build.BuildId)

trigger:
 branches:
   include:
     - master/TestFixtureInitializer

variables:
- group: 'PipelineVariables'

pool: TestAutomationExamples

jobs:
- job: default
  timeoutInMinutes: 240
  pool: TestAutomationExamples

  steps:
  - checkout: self
    clean: true 

  - task: ALOpsDockerCreate@1
    displayName: 'ALOPS - Create Docker Image' # It will reuse the image if already exists, or build one if it doesn't.
    inputs:
      forcecreateimage: $(forceCreateImage)
      licensefile: '$(LuxBeheerBcDevLicense)'

  - task: ALOpsDockerStart@1
    displayName: 'ALOPS - Start Docker Container' # No need to provide any details - it will get the details from previous step

  - task: ALOpsDockerWait@1
    displayName: 'ALOPS - Wait for Docker Container to start'
    inputs:
      search_string: 'Ready for connections!'

  - task: ALOpsDockerExec@1
    displayName: 'Uninstall MS apps (but leave Application apps)'
    inputs:
      inline_script: |
        Get-NAVAppInfo -ServerInstance BC -tenant default -tenantspecificproperties | 
          where isinstalled -eq $true | 
            Where Publisher -eq Microsoft | 
              where {$_.Name -ne 'Base Application' -and $_.Name -ne 'System Application' -and $_.Name -ne 'Application'}  | 
                uninstall-navapp -tenant default -force

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Install AL TestTool' # install necessary dependent apps for testability
    inputs:
      usedocker: true
      installaltesttool: true
      skip_verification: true
      install_al_app_names: |
        Any
        Library Assert
        Library Variable Storage
        System Application Test Library
        Tests-TestLibraries
        Tests-Cost Accounting
        Tests-CRM integration
        Test Runner

  - task: ALOpsAppCompiler@1
    displayName: 'ALOPS - Compile Extension: App'
    inputs:
      usedocker: true
      nav_app_version: '1.0.[yyyyWW].*'
      targetproject: 'TestFixtureInitializer/app/app.json'
      app_file_suffix: '_APP'                                 # A trick to uniquely identify the APP (from the TEST-app)

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Publish Extension'
    inputs:
      usedocker: true
      nav_artifact_app_filter: '*.app'
      skip_verification: true

  - task: ALOpsAppCompiler@1
    displayName: 'ALOPS - Compile Extension: Test-App'
    inputs:
      usedocker: true
      nav_app_version: '1.0.[yyyyWW].*'
      targetproject: 'TestFixtureInitializer/test/app.json'
      failed_on_warnings: true    
      app_file_suffix: '_TEST'                                 # A trick to uniquely identify the Test-App (from the App)

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Publish Extension'
    inputs:
      usedocker: true
      nav_artifact_app_filter: '*.app'
      skip_verification: true   

  - task: ALOpsAppTest@1
    displayName: 'ALOPS - Run Tests'
    inputs:
      usedocker: true
      disabledtests: '$(Build.SourcesDirectory)\TestFixtureInitializer\Test\DisabledTests\DisabledTests-Batch1.json' # See https://github.com/fluxxus-nl/TestFixtureInitializer/issues/4
      import_testtoolkit: false
      import_action: 'Skip'
      testpage: '130455'
      testsuite: 'DEFAULT'
      failed_test_action: 'Ignore'
    continueOnError: true
    timeoutInMinutes: 180

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifact Test Results **/TestResults.xml'
    inputs:
      targetPath: '$(System.DefaultWorkingDirectory)/TestResults.xml'
      artifact: 'TestResults.xml'
      publishLocation: 'pipeline'

  - task: PublishTestResults@2
    displayName: 'Publish Test Results **/TestResults.xml'
    inputs:
      testResultsFormat: XUnit
      testResultsFiles: '**/TestResults.xml'
      failTaskOnFailedTests: true
      testRunTitle: 'BC Test Results: $(Build.BuildId)'

  - task: ALOpsDockerRemove@1
    displayName: 'ALOPS - Remove Docker Container'
    enabled: true
    condition: always()
    inputs:
      createsqlbackup: '$(createSqlBackup)'

[waldo1001]

Nice. Thanks! No need to create new issue - extensionid is on the table. We'll need it going further anyway as well (v18).

I also like to work with the install codeunits - it gives a certain amount of flexibility which is going to be difficult in pure yaml..

[lvanvugt]

Code, .yml and DisabledTests,json can be found here: https://github.com/fluxxus-nl/TestFixtureInitializer/

[jfamvg]

@waldo1001 @AdminHodor I faced the same issue as lvanvugt, I want to run all the tests from the compiled & installed apps. I noticed that the runner will load all test CU's when I open page 130455 in BC and set the CU filter like '<> 0'. This page is also set as the testpage for the ALOpsAppTest task. Therefore I'm wondering, did something change since this issue was closed? Is it now technically possible to load codeunits with the filter?

Thanks in advance!

[waldo1001]

@jfamvg , we didn't change anything, so if it works differently, then it's MS changing stuff.

I can't really follow what you mean (we don't really do fancy stuff with filtering tests ;-)), so I'd suggest to open a new issue (if you have an issue ;-)), with enough details?