Question: How to let ALOps with multiple ALOpsAppTest steps create one aggregated TestResults.xml?

[lvanvugt]

In issue #310 I asked

what is the best/efficient way to populated a test suite with all tests present?

Due to that question and the discussion following it, you have added a new feature to ALOps (in upcoming release). As you can read there new info from MS made clear this will probably not going to work right. MS has bucketed there test runs, meaning that most tests are run in different buckets/runs. It's similar (not the same) to what I have done on my TestFixtureInitializer project. In the latter project each MS test app has it's own bucket (i.e. build pipeline); 31 at this very moment. Now, I wanted to experiment to see if I could enlarge my buckets by combining a number of test apps into one pipeline, even though each test app tests will be run in it's own AL Test Tool test suite. See the pipeline yaml below, where each test app test run will be triggered by a dedicated ALOpsAppTest step (in case of this pipeline three times the ALOpsAppTest step). Test are indeed executed, but the resulting TestResults.xml fetched and published by the PublishTestResults step does only contain the results of the last test run.

Question

Is there a way to combine the test results of all ALOpsAppTest steps into one TestResults.xml?

the used yaml

name: $(Build.BuildId)

trigger:
 branches:
   include:
     - master/TestFixtureInitializer

variables:
- group: 'PipelineVariables'

pool: TestAutomationExamples

jobs:
- job: default
  timeoutInMinutes: 600
  pool: TestAutomationExamples

  steps:
  - checkout: self
    clean: true 

  - task: ALOpsDockerCreate@1
    displayName: 'ALOPS - Create Docker Image' # It will reuse the image if already exists, or build one if it doesn't.
    inputs:
      forcecreateimage: $(forceCreateImage)
      licensefile: '$(LuxBeheerBcDevLicense)'

  - task: ALOpsDockerStart@1
    displayName: 'ALOPS - Start Docker Container' # No need to provide any details - it will get the details from previous step

  - task: ALOpsDockerWait@1
    displayName: 'ALOPS - Wait for Docker Container to start'
    inputs:
      search_string: 'Ready for connections!'

  - task: ALOpsDockerExec@1
    displayName: 'Uninstall MS apps (but leave Application apps)'
    inputs:
      inline_script: |
        Get-NAVAppInfo -ServerInstance BC -tenant default -tenantspecificproperties | 
          where isinstalled -eq $true | 
            Where Publisher -eq Microsoft | 
              where {$_.Name -ne 'Base Application' -and $_.Name -ne 'System Application' -and $_.Name -ne 'Application'}  | 
                uninstall-navapp -tenant default -force

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Install AL TestTool' # install necessary dependent apps for testability
    inputs:
      usedocker: true
      installaltesttool: true
      skip_verification: true
      install_al_app_names: |
        Any
        Library Assert
        Library Variable Storage
        System Application Test Library
        Tests-TestLibraries
        Tests-Cash Flow
        Tests-Cost Accounting
        Tests-CRM integration
        Test Runner

  - task: ALOpsAppCompiler@1
    displayName: 'ALOPS - Compile Extension: App'
    inputs:
      usedocker: true
      nav_app_version: '1.0.[yyyyWW].*'
      targetproject: 'TestFixtureInitializer/app/app.json'
      app_file_suffix: '_APP'                                 # A trick to uniquely identify the APP (from the TEST-app)

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Publish Extension'
    inputs:
      usedocker: true
      nav_artifact_app_filter: '*.app'
      skip_verification: true

  - task: ALOpsAppCompiler@1
    displayName: 'ALOPS - Compile Extension: Test-App'
    inputs:
      usedocker: true
      nav_app_version: '1.0.[yyyyWW].*'
      targetproject: 'TestFixtureInitializer/test/app.json'
      failed_on_warnings: true    
      app_file_suffix: '_TEST'                                 # A trick to uniquely identify the Test-App (from the App)

  - task: ALOpsAppPublish@1
    displayName: 'ALOPS - Publish Extension'
    inputs:
      usedocker: true
      nav_artifact_app_filter: '*.app'
      skip_verification: true   

  - task: ALOpsAppTest@1
    displayName: 'ALOPS - Run TestSuite CASH FLOW'
    inputs:
      usedocker: true
      import_action: "Skip"
      import_testtoolkit: false
      testpage: '130455'
      testsuite: 'CASH FLOW'
      failed_test_action: 'Ignore'
    continueOnError: true
    timeoutInMinutes: 45

  - task: ALOpsAppTest@1
    displayName: 'ALOPS - Run TestSuite COST ACCOUNTING'
    inputs:
      usedocker: true
      import_action: "Skip"
      import_testtoolkit: false
      testpage: '130455'
      testsuite: 'COST ACCOU'
      failed_test_action: 'Ignore'
    continueOnError: true
    timeoutInMinutes: 45

  - task: ALOpsAppTest@1
    displayName: 'ALOPS - Run TestSuite CRM INTEGRATION'
    inputs:
      usedocker: true
      import_action: "Skip"
      import_testtoolkit: false
      testpage: '130455'
      testsuite: 'CRM INTEGR'
      failed_test_action: 'Ignore'
    continueOnError: true
    timeoutInMinutes: 45

  - task: PublishTestResults@2
    displayName: 'Publish Test Results **/TestResults.xml'
    inputs:
      testResultsFormat: XUnit
      testResultsFiles: '**/TestResults.xml'
      failTaskOnFailedTests: true
      testRunTitle: 'BC Test Results: $(Build.BuildId)'

  - task: ALOpsDockerRemove@1
    displayName: 'ALOPS - Remove Docker Container'
    enabled: true
    condition: always()
    inputs:
      createsqlbackup: '$(createSqlBackup)'

Additional discussion I reckon the feature you have added because of #310 will allow me to get this done too, but if the above way also works I could be able to combine different bucket runs with clean docker containers consecutively (in one pipeline) and aggregate the results.

[lvanvugt]

Note that BcContainerHelper cmdlet Run-TestsInBcContainer allows you to append to the TestResults.xml created by a previous call to the same cmdlet.

[lvanvugt]

@AdminHodor, I am wondering why there is no response yet. I might be not clear enough. If so, let me know.

[waldo1001]

I did respond through mail (with Freddy).

But forgot to flip the switch to "part of upcoming release" - it's been on the roadmap since last week .. but didn't update the status ;-).

[lvanvugt]

You might want to include that discussion here as a (first kind of) documentation.

[lvanvugt]

@waldo1001 any update on this?

[waldo1001]

As you noticed, we have not the ability to have multiple test runs, which "aggregates" in one output in DevOps.

Do we assume right that this is enough? Is there any reason why they should be joined?

[lvanvugt]

Do we assume right that this is enough?

Maybe I misunderstand this question, but my answer now is: no, as it is not enough. That's the basis to this issue.

Is there any reason why they should be joined?

To have one result telling me that a collection of tests are successful and not each individual test app.

But for now we might just park this as MS test collateral is not able to be run easily in one stretch.

[waldo1001]

Thing is, the entire testability-framework is getting an overhaul. I'd want to limit unnecessary investment, if you know what I mean ;-). If you're ok to close this issue, I'm more than ok as well ;-)

[lvanvugt]

Any progress on this?

[waldo1001]

Well, MS is very quiet on anything Testability-related changes. It seems that the big overhaul isn't that big after all, and I don't know what to expect in terms of this.. .

I'll put it back on the list, since the only info that I got of changing test-behavior was NOT in terms of "running tests" from powershell.

[lvanvugt]

Any progress on this?

[waldo1001]

None at all. As this was agreed to be low prio, we tend to focus on other things.

Can you explain a business case on how this would benefit us? We simply never concatenate, and still have all tests available in our pipelines output.. ?

[lvanvugt]

Business case: running different tests apps in one pipeline one after the other not loading the test of all apps into one suite. This way you have a better overview in which test a a failure happens. The tests results tab in the pipeline portal does not give me that insight. In between the different test runs I would also be able to remove the previous run test app and install the next one.

[lvanvugt]

@waldo1001, this issue has now become more important. Have a look here: https://github.com/PacktPublishing/Automated-Testing-in-Microsoft-Dynamics-365-Business-Central-Second-Edition/issues/16#issuecomment-1138298015.

[waldo1001]

I'm trying to see why this is now more important, Luc .. .

Here is an example of running tests of 41 apps, with an out-of-the-devops-box "aggregated" result:

or including succeeded tests:

Can you please explain what data I'm missing in this overview?

[lvanvugt]

The more important is related to that I am now in need of getting the results aggregated.

Nevertheless, how do you get your results aggregated. Apparently (seeing the screenshots above and looking great) you have a way, right? So, what should I do to achieve this?

[waldo1001]

We're not doing anything special, simply:

• install-codeunit that creates a unique testsuite per app
• have all those apps installed
• run all suites one-by-one:

• And after each run, publish the testresult (probably that's the "tip" to make this work):

[lvanvugt]

• And after each run, publish the testresult (probably that's the "tip" to make this work):

Simplicity counts. I guess I am to "intelligent" for that. LOL

[waldo1001]

Let me know if this works for you ;-)

[lvanvugt]

Meanwhile found a solution to merge the multiple test result xml files into one and publish that alone. Works like a charm (too).

- task: ALOpsAppTest@1
  displayName: 'ALOPS - Run TestSuite 1
  inputs:
    ...
    resultfilename: 'TestResults1.xml'
  continueOnError: true

- task: ALOpsAppTest@1
  displayName: 'ALOPS - Run TestSuite 2'
  inputs:
    ...
    resultfilename: 'TestResults2.xml'
  continueOnError: true

- task: PowerShell@2
  displayName: 'Merge test Results'
  inputs:
    targetType: 'inline'
    script: |
      $TestResults1 = [xml](Get-Content -Path $(Build.SourcesDirectory)/TestResults1.xml)
      $TestResults2 = [xml](Get-Content -Path $(Build.SourcesDirectory)/TestResults2.xml)
      $TestResults2.DocumentElement.ChildNodes | ForEach-Object { $TestResults1.DocumentElement.AppendChild($TestResults1.ImportNode($_, $true)) | Out-Null }
      Set-Content -Path $(Build.SourcesDirectory)/TestResults.xml -Value ($TestResults1.OuterXml)

- task: PublishTestResults@2
  displayName: 'Publish Test Results **/TestResults.xml'
  inputs:
    testResultsFormat: XUnit
    testResultsFiles: '**/TestResults.xml'
    failTaskOnFailedTests: true
    testRunTitle: 'BC Test Results: $(Build.BuildId)'

[lvanvugt]

Let me know if this works for you ;-)

Will keep that in mind. Does not meet my requirements right now. Will, however, close the issue.