search

HodorNV / ALOps

ALOps
59 stars 24 forks source link

Pipeline successfully finishes despite failed auto test(s) #521

Closed Solmyr1982 closed 2 years ago

Solmyr1982 commented 2 years ago

Describe the bug When running the deployment pipeline, step Run TestSuite, some tests fail. It's noticeable from the log: Codeunit 54986 POS Development Testsuite IMV Failure Despite this, the pipeline ignores the failure and continues running, ending up successfully.

the used yaml


    - task: ALOpsAppTest@1
      timeoutInMinutes: 30
      displayName: "Run TestSuite"
      inputs:
        usedocker: true
        testfilter: "(50000..70099|70301..99999)&<>53051"
        import_testtoolkit: false
        show_available_tests: true
        failed_test_action: Error
        testpage: 90409
    - task: PublishTestResults@2
      displayName: "Publish Test Results **/TestResults.xml"
      condition: succeededOrFailed()
      inputs:
        testResultsFormat: XUnit
        testResultsFiles: "**/TestResults.xml"
        testRunTitle: "BC Test Results: $(Build.BuildId)"

the output


##[section]Starting: Run TestSuite
==============================================================================
Task         : ALOps App Test
Description  : Run AL-TestSuite for Business Central
Version      : 1.448.2984
Author       : Hodor
Help         : Run Business Central Test-Suite and collect results.
==============================================================================
*** Validate configuration
*** Task Inputs:

name                                               value
----                                               -----
usedocker                                           True
fixed_tag                                               
nav_serverinstance                                 BC140
tenant                                           default
companyname                                             
profile                                                 
bc_username                                             
bc_password                                             
bc_authentication                        NavUserPassword
bc_webclient_url                                        
testfilter           (50000..70099|70301..99999)&<>53051
disabledtests                                           
show_available_tests                                True
import_testtoolkit                                 False
import_action                                  Overwrite
testpage                                           90409
testsuite                                        DEFAULT
extensionid                                             
installaltesttool                                  False
failed_test_action                                 Error
override_finsql_path                                    
resultfilename                           TestResults.xml

*** For documentation, please visit   : https://www.alops.be/documentation

*** ALOps License:
  * Licensed To: K3 Business Solutions (Project License)

*** Importing required PS-Functions
*** Working folder: C:\a\2\s
***
*** Prepare Docker Session
*** Connect Docker Session
*** Initiate Docker Session
*** Set Docker Container ErrorActionPreference = Stop
*** Copy Modules to Docker
*** Setup Docker Session
*** Checking for container-specific Test Assemblies
*** Using container-specific Test Assemblies
*** Import Modules in Docker Session
*** Import UI Client [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
VERBOSE: Loading module from path 'C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll'.
*** Import NewtonSoft JSON [C:\Program Files\Microsoft Dynamics NAV\140\Service\Newtonsoft.Json.dll]
*** Import Modules in Docker Session: class_ClientContext
*** Test Context Url: https://d5a521be3929/NAV/cs?tenant=default
*** Setup BC Credentials in Docker
*** Resolve Docker Credentials
*** Set TCP Keep-Alive
*** Start TestSuite
*** Running TestSuite with authentication [NavUserPassword] with User [admin]
*** Run warm-up with page [90409]
*** Warm-up Client Context
*** Create Client Context
*** Add Tests to Default-TestSuite with filter [(50000..70099|70301..99999)&<>53051]
Show form {00016129-0000-0000-0108-0000836BD2D2}
Title: MV Test Tool Page Edit - Test Tool - DEFAULT
Apply filter for Test-Codeunits: [(50000..70099|70301..99999)&<>53051]
*** Get Available Tests for suite [DEFAULT] with page [90409]
Show form {00016129-0000-0000-0108-0000836BD2D2}
Title: MV Test Tool Page Edit - Test Tool - DEFAULT
*** Set [CurrentSuiteName] => [DEFAULT]
{
    "Group":  "Default Suite - Autogenerated",
    "Codeunits":  [
                      {
                          "Tests":  "TestPostCodeUsed TestGeoLocationsImport",
                          "Name":  "Geographic API TestSuite 1 IMV",
                          "Id":  "54775"
                      },
                      {
                          "Tests":  "OnRun OpenReport54792 OpenReport54793 OpenReport54794 OpenReport54795 OpenReport54796 OpenReport54797 OpenReport54798 OpenReport54799",
                          "Name":  "LocalAddressChgTestSuiteIMV",
                          "Id":  "54810"
                      },
                      {
                          "Tests":  "TestCollectionPointsImport UpdateShipToCityFromSellToCity",
                          "Name":  "Nodes API TestSuite IMV",
                          "Id":  "54812"
                      },
                      {
                          "Tests":  "TestFieldsValidation TestServiceOrderItemsServiceOrderNoValidation TestServiceTimeSlotsReservationAPI TestIKEASalesOrderPaymentDueDateTime TestServiceOrderDisablePickingService OpenPage54874Wizard IKEAAfterSalesOrderWithItemsOpenCLServiceWizard IKEAEmptySalesOrderOpenCLServiceWizard IKEASalesOrderWithItemsOpenCLServiceWizard",
                          "Name":  "Shipping Aggr. TestSuite IMV",
                          "Id":  "54833"
                      },
                      {
                          "Tests":  "TestNewFields TestFieldsEditability",
                          "Name":  "Fullfil. Ord. Test Suite 1 IMV",
                          "Id":  "54853"
                      },
                      {
                          "Tests":  "TestNewFields CreateEcomOrderFromAPIBuffer CreateEcomOrderFromAPIBufferWithNewContact RemovePaymentDateTimeOnCancelOrder CheckPaymentOption AutomaticallyCreateEcomStatement",
                          "Name":  "Ecom API TestSuite1 IMV",
                          "Id":  "54892"
                      },
                      {
                          "Tests":  "TestMixNormalAndNonSaleItems TestCreateCustomerPOSInfoAnd11Check TestCreditNoteTransactionWithPOSInfo TestReturnTransactionDetails TestNationalIDToCustomerOnTotal TestReturnReceptWithPOSInfo CreateFloatEntryWithNoFiscalEntry CreateRemoveTenderEntryWithNoFiscalEntry CancellationOrder TestNewRecoverySetupFields TestNewRecoveryJnlLineFields",
                          "Name":  "POS Development Testsuite IMV",
                          "Id":  "54986"
                      },
                      {
                          "Tests":  "TestFPaySetupFields CreateFPayEntry CreateFPayTenderType OpenPage54990 OpenPage54992 OpenPage55995 CreateTransFPayEntry",
                          "Name":  "FPay TestSuite IMV",
                          "Id":  "54990"
                      },
                      {
                          "Tests":  "OpenPage460 OpenPage54988",
                          "Name":  "POS Dev Pages Testsuite IMV",
                          "Id":  "54996"
                      },
                      {
                          "Tests":  "CreateFiscalTransForDeliveryNoteFromHoldingJnl",
                          "Name":  "Delivery Note Testsuite IMV",
                          "Id":  "54999"
                      },
                      {
                          "Tests":  "TestPaymentMeansCode TestRutNumber TestFiscalDocNoSeries TestPOSTerminalCreated TestPOSTerminalDeleted TestFiscalDocMappingSetupCreated TestWSSetups TestRetrieveFoliosNeeded TestCloseExpiredFiscalDocNoSeries TestTransactionWithFiscalEntry TestRetrieveFoliosDisable TestTransactionWithFiscalEntryReturn TestDebitNote OpenPage55990 OpenPage55991 OpenPage55992",
                          "Name":  "DTE Mgt Test Suite 1 IMV",
                          "Id":  "55990"
                      },
                      {
                          "Tests":  "TestPurchOrderLineInsert TestUpdatePurchaseOrderQuantityNothingToUpdate TestUpdatePurchaseOrderQuantity TestPurchaseOrderDeletion TestPurchOrderReceipt TestReleasePurchaseOrder TestReleasePurchaseOrderDataExport LocalSuppliersOnSendForApprovalOfPurchaseOrderPlants DefaultLocalSupplierFilledForPurchaseOrderPlantsItems OpenPage59271 OpenPage59272 TestInsertItemPlantsWithoutContractID",
                          "Name":  "Local Purchasing TestSuite IMV",
                          "Id":  "59268"
                      },
                      {
                          "Tests":  "TestCashierShift",
                          "Name":  "TAE TestSuite IMV",
                          "Id":  "59317"
                      }
                  ]
}
*** Run Available Tests for suite [DEFAULT] with page [90409]
*** Connect TestClient with page [90409]
Show form {00016129-0000-0000-0108-0000836BD2D2}
Title: MV Test Tool Page Edit - Test Tool - DEFAULT
  Codeunit 54775 Geographic API TestSuite 1 IMV Success (6.852 seconds)
  Codeunit 54810 LocalAddressChgTestSuiteIMV Success (2.868 seconds)
  Codeunit 54812 Nodes API TestSuite IMV Success (2.094 seconds)
  Codeunit 54833 Shipping Aggr. TestSuite IMV Success (9.606 seconds)
  Codeunit 54853 Fullfil. Ord. Test Suite 1 IMV Success (4.269 seconds)
  Codeunit 54892 Ecom API TestSuite1 IMV Success (8.955 seconds)
  Codeunit 54986 POS Development Testsuite IMV Failure (9.15 seconds)
False
*** XUnit Test results: c:\Run\DevOps\TestResults.xml
<?xml version="1.0" encoding="UTF-8"?>
<assemblies>
  <assembly name="54775 Geographic API TestSuite 1 IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:14" total="2" passed="2" failed="0" skipped="0" time="6.223">
    <collection name="Geographic API TestSuite 1 IMV" total="2" passed="2" failed="0" skipped="0" time="6.223" Skipped="0">
      <test name="Geographic API TestSuite 1 IMV:TestPostCodeUsed" method="TestPostCodeUsed" time="5.927" result="Pass" />
      <test name="Geographic API TestSuite 1 IMV:TestGeoLocationsImport" method="TestGeoLocationsImport" time="0.296" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54810 LocalAddressChgTestSuiteIMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:21" total="9" passed="9" failed="0" skipped="0" time="2.389">
    <collection name="LocalAddressChgTestSuiteIMV" total="9" passed="9" failed="0" skipped="0" time="2.389" Skipped="0">
      <test name="LocalAddressChgTestSuiteIMV:OnRun" method="OnRun" time="0" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54792" method="OpenReport54792" time="1.733" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54793" method="OpenReport54793" time="0.047" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54794" method="OpenReport54794" time="0.063" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54795" method="OpenReport54795" time="0.093" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54796" method="OpenReport54796" time="0.063" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54797" method="OpenReport54797" time="0.233" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54798" method="OpenReport54798" time="0.08" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54799" method="OpenReport54799" time="0.077" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54812 Nodes API TestSuite IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:24" total="2" passed="2" failed="0" skipped="0" time="1.66">
    <collection name="Nodes API TestSuite IMV" total="2" passed="2" failed="0" skipped="0" time="1.66" Skipped="0">
      <test name="Nodes API TestSuite IMV:TestCollectionPointsImport" method="TestCollectionPointsImport" time="1.613" result="Pass" />
      <test name="Nodes API TestSuite IMV:UpdateShipToCityFromSellToCity" method="UpdateShipToCityFromSellToCity" time="0.047" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54833 Shipping Aggr. TestSuite IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:26" total="9" passed="9" failed="0" skipped="0" time="9.015">
    <collection name="Shipping Aggr. TestSuite IMV" total="9" passed="9" failed="0" skipped="0" time="9.015" Skipped="0">
      <test name="Shipping Aggr. TestSuite IMV:TestFieldsValidation" method="TestFieldsValidation" time="1.697" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:TestServiceOrderItemsServiceOrderNoValidation" method="TestServiceOrderItemsServiceOrderNoValidation" time="1.063" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:TestServiceTimeSlotsReservationAPI" method="TestServiceTimeSlotsReservationAPI" time="0.236" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:TestIKEASalesOrderPaymentDueDateTime" method="TestIKEASalesOrderPaymentDueDateTime" time="0.204" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:TestServiceOrderDisablePickingService" method="TestServiceOrderDisablePickingService" time="3.063" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:OpenPage54874Wizard" method="OpenPage54874Wizard" time="2.346" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:IKEAAfterSalesOrderWithItemsOpenCLServiceWizard" method="IKEAAfterSalesOrderWithItemsOpenCLServiceWizard" time="0.236" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:IKEAEmptySalesOrderOpenCLServiceWizard" method="IKEAEmptySalesOrderOpenCLServiceWizard" time="0.03" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:IKEASalesOrderWithItemsOpenCLServiceWizard" method="IKEASalesOrderWithItemsOpenCLServiceWizard" time="0.14" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54853 Fullfil. Ord. Test Suite 1 IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:36" total="2" passed="2" failed="0" skipped="0" time="3.783">
    <collection name="Fullfil. Ord. Test Suite 1 IMV" total="2" passed="2" failed="0" skipped="0" time="3.783" Skipped="0">
      <test name="Fullfil. Ord. Test Suite 1 IMV:TestNewFields" method="TestNewFields" time="1.94" result="Pass" />
      <test name="Fullfil. Ord. Test Suite 1 IMV:TestFieldsEditability" method="TestFieldsEditability" time="1.843" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54892 Ecom API TestSuite1 IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:40" total="6" passed="6" failed="0" skipped="0" time="8.41">
    <collection name="Ecom API TestSuite1 IMV" total="6" passed="6" failed="0" skipped="0" time="8.41" Skipped="0">
      <test name="Ecom API TestSuite1 IMV:TestNewFields" method="TestNewFields" time="6.483" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:CreateEcomOrderFromAPIBuffer" method="CreateEcomOrderFromAPIBuffer" time="1.047" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:CreateEcomOrderFromAPIBufferWithNewContact" method="CreateEcomOrderFromAPIBufferWithNewContact" time="0.486" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:RemovePaymentDateTimeOnCancelOrder" method="RemovePaymentDateTimeOnCancelOrder" time="0.284" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:CheckPaymentOption" method="CheckPaymentOption" time="0.093" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:AutomaticallyCreateEcomStatement" method="AutomaticallyCreateEcomStatement" time="0.017" result="Pass" />
    </collection>
  </assembly>
</assemblies>

*** Get XUnit Test file from [c:\Run\DevOps\TestResults.xml] to [C:\a\2\s\TestResults.xml]

*** Disconnect Docker session.
*** Docker session removed.
*** Running Tests Completed.
*** Cleanup VSTS Environment: True
##[section]Finishing: Run TestSuite

Expected behavior We expect that the pipeline execution will break, informing the user of the failed auto test.

Screenshots image

image

Additional context This behavior changes when I renumber the test codeunit 54892 to 59340. After that the pipeline breaks on the failed tests. Both codeunits are within the license.

@rppwolfs

waldo1001 commented 2 years ago

What version of BC?

Solmyr1982 commented 2 years ago

It's BC 14: image

rppwolfs commented 2 years ago

BC14 (CU24 April 2019 / Platform Build 14.0.47286).

waldo1001 commented 2 years ago

Oh .. ok. I'm afraid it's version-related - but I'll see if we can capture it nevertheless.. (this "test run" isn't the most flexible thing to use ;-))

NAVFreak commented 2 years ago

I'm having the same problem but I run the latest version The goal is accept warnings in the building process, create and publish testresults but fail the pipeline if all the tests didnt succeed. Since I couldn't get to publish the test results when the test failed I tried make an "exception handling" that it always publishes the tests and then fail the pipeline if the running of tests failed. But it seems it doesnt fail.

##[section]Starting: ALOPS - Run TestSuite

Task         : ALOps App Test
Description  : Run AL-TestSuite for Business Central
Version      : 1.449.3024
Author       : Hodor
Help         : Run Business Central Test-Suite and collect results.

*** Validate configuration
*** Task Inputs:

name                           value
----                           -----
usedocker                       True
fixed_tag
nav_serverinstance             BC140
tenant                       default
companyname
profile
bc_username
bc_password
bc_authentication    NavUserPassword
bc_webclient_url
testfilter
disabledtests
show_available_tests            True
import_testtoolkit             False
import_action                   Skip
testpage                      130455
testsuite                    DEFAULT
extensionid
installaltesttool              False
failed_test_action             Error
override_finsql_path
resultfilename       TestResults.xml

*** For documentation, please visit   : https://www.alops.be/documentation

*** ALOps License:
* Licensed To: Navet AB (Collection/Organization License)

*** Importing required PS-Functions
*** Working folder: c:\vsts-agent-win-x64-2.202.0\_work\3\s
***
*** Prepare Docker Session
*** Connect Docker Session
*** Initiate Docker Session
*** Set Docker Container ErrorActionPreference = Stop
*** Copy Modules to Docker
*** Setup Docker Session
*** Checking for container-specific Test Assemblies
*** Using container-specific Test Assemblies
*** Import Modules in Docker Session
*** Import UI Client [C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll]
VERBOSE: Loading module from path 'C:\Run\DevOps\BCAppTest\bin\Microsoft.Dynamics.Framework.UI.Client.dll'.
*** Import NewtonSoft JSON [C:\Program Files\Microsoft Dynamics NAV\200\Service\Newtonsoft.Json.dll]
*** Import Modules in Docker Session: class_ClientContext
*** Test Context Url: https://681435b7d720/BC/cs?tenant=default
*** Setup BC Credentials in Docker
*** Resolve Docker Credentials
*** Set TCP Keep-Alive
*** Start TestSuite
*** Running TestSuite with authentication [NavUserPassword] with User [admin]
*** Run warm-up with page [130455]
*** Warm-up Client Context
*** Create Client Context
*** Get Available Tests for suite [DEFAULT] with page [130455]
Show form 0001fd97-0000-0001-0c00-0000836bd2d2
Title: Command Line Test Tool Edit - Command Line Test Tool - DEFAULT
*** Set [CurrentSuiteName] => [DEFAULT]
[
{
"Tests":  [
"ReleaseSecondPartOrder"
],
"Name":  "NBAWMSSalesPartOrderTest",
"Id":  "99000"
},
{
"Tests":  [
"DeliverCompleteOrder",
"PartDeliverOrder",
"BelowDeliverOrderMakeAdjustment",
"BelowDeliverOrderDontMakeAdjustment",
"OverDeliverOrder",
"BlockOverDeliverOrder",
"OverDelivery2nd_ExactMode",
"OverDelivery2nd_CumulativedMode",
"LastDoubleDelivery_CumulativedMode",
"CompletedOrderMarkedAsFinished"
],
"Name":  "NBAWMSPurchaseOrderReceiveTest",
"Id":  "99004"
},
{
...
"Tests":  [
"TestTransferAllToItemJnl",
"TestPostAllInItemJnl"
],
"Name":  "NBAWMS Stock Report Test",
"Id":  "99008"
}
]
*** Run Available Tests for suite [DEFAULT] with page [130455]
*** Connect TestClient with page [130455]
Show form 0001fd97-0000-0001-0c00-0000836bd2d2
Title: Command Line Test Tool Edit - Command Line Test Tool - DEFAULT
Codeunit 99000 NBAWMSSalesPartOrderTest Failure (20.301 seconds)
Testfunction ReleaseSecondPartOrder Failure (18.583 seconds)
Codeunit 99004 NBAWMSPurchaseOrderReceiveTest Success (19.825 seconds)
Codeunit 99005 NBAWMSSalesOrderDeliverTest Success (8.282 seconds)
Codeunit 99006 NBAWMSSalesRetOrderReceiveTest Success (5.04 seconds)
Codeunit 99007 NBAWMSPurchaseRetOrderDeliver Success (6.544 seconds)
Codeunit 99008 NBAWMS Stock Report Test Success (3.936 seconds)
False
*** XUnit Test results: c:\Run\DevOps\TestResults.xml
<?xml version="1.0" encoding="UTF-8"?>
<assemblies>
<assembly name="99000 NBAWMSSalesPartOrderTest" test-framework="PS Test Runner" run-date="2022-06-10" run-time="07:21:12" total="1" passed="0" failed="1" skipped="0" time="18.583">
<collection name="NBAWMSSalesPartOrderTest" total="1" passed="0" failed="1" skipped="0" time="18.583" Skipped="0">
<test name="NBAWMSSalesPartOrderTest:ReleaseSecondPartOrder" method="ReleaseSecondPartOrder" time="18.583" result="Fail">
<failure>
<message>We could not update the data right now.\Please try again later.\\Technical details:\A record in table Sales Line was locked for use in another session. Administrators can see current locks on the Database Locks page or use telemetry. (Snapshot ID: Failed to retrieve relevant parameter)</message>
<stack-trace>"Library - Sales"(CodeUnit 130509).CreateSalesLineWithShipmentDate line 23 - Tests-TestLibraries by Microsoft
"Library - Sales"(CodeUnit 130509).CreateSalesLine line 2 - Tests-TestLibraries by Microsoft
"Library - Sales"(CodeUnit 130509).CreateSalesOrder line 8 - Tests-TestLibraries by Microsoft
"NBAWMS Helper Functions"(CodeUnit 99003).CreateSalesOrder line 9 - Test Navet WMS by Navet AB
NBAWMSSalesPartOrderTest(CodeUnit 99000).ReleaseSecondPartOrder line 10 - Test Navet WMS by Navet AB
"Test Runner - Mgt"(CodeUnit 130454).RunTests line 18 - Test Runner by Microsoft
"Test Runner - Isol. Codeunit"(CodeUnit 130450).OnRun(Trigger) line 4 - Test Runner by Microsoft
"Test Suite Mgt."(CodeUnit 130456).RunTests line 2 - Test Runner by Microsoft
"Test Suite Mgt."(CodeUnit 130456).RunSelectedTests line 35 - Test Runner by Microsoft
"Command Line Test Tool"(Page 130455)."RunSelectedTests - OnAction"(Trigger) line 7 - Test Runner by Microsoft</stack-trace>
</failure>
</test>
</collection>
</assembly>
...
</assemblies>

*** Get XUnit Test file from [c:\Run\DevOps\TestResults.xml] to [c:\vsts-agent-win-x64-2.202.0\_work\3\s\TestResults.xml]

*** Disconnect Docker session.
*** Docker session removed.
##[warning]NBAWMSSalesPartOrderTest:ReleaseSecondPartOrder: We could not update the data right now.\Please try again later.\\Technical details:\A record in table Sales Line was locked for use in another session. Administrators can see current locks on the Database Locks page or use telemetry. (Snapshot ID: Failed to retrieve relevant parameter)
##[error]*** Running Tests failed.
Id Name   ComputerName                                                     ComputerType State  ConfigurationName Availa
bility
-- ----   ------------                                                     ------------ -----  ----------------- ------
1 WinRM1 681435b7d720fe94c621d7ba15c7fca5b40e99f473afda88cb22db1a172aea9b Container    Opened                   ...ble

##[section]Finishing: ALOPS - Run TestSuite

My yaml consists of three stages:

Here is how the test stage look:

  - stage: Test
    dependsOn: Build
    jobs:
      - job: RunTests
        steps:
          - template: ./DevOps/Compile_And_Publish.yml@Templates
            parameters:
              app_name: 'Test APP'
              bc_app_version: '1.0.[yyyyWW].*'
              app_file_suffix: '_TEST'
              failed_on_warnings: false
              targetproject: 'test/app.json'

          - task: ALOpsAppTest@1
            displayName: 'ALOPS - Run TestSuite'
            inputs:
              usedocker: true
              import_action: "Skip"
              import_testtoolkit: false
              testpage: '130455'
              testsuite: 'DEFAULT'
              failed_test_action: Error   #Warning/Error
            continueOnError: true    # <------------------- This setting decides if the build should fail if the tests fail   

          - task: PublishTestResults@2
            displayName: 'Publish Test Results **/TestResults.xml'
            condition: always()
            inputs:
              testResultsFormat: XUnit
              testResultsFiles: '**/TestResults.xml'
              testRunTitle: 'BC Test Results: $(Build.BuildId)'

      - job: FailIfTestsFailed
        dependsOn: RunTests
        #condition: and(failed('RunTests'),eq(variables['Agent.JobStatus'], 'SucceededWithIssues'))
        # condition: and(failed('RunTests'),eq(variables['Agent.JobStatus'], 'SucceededWithIssues'))
        condition: failed('RunTests')
        steps:
          - bash: exit 1
            displayName: Fail build if tests partially successful

  - stage: Cleanup
    condition: always()
    jobs:
      - job: RemoveDocker
        steps:
          - task: ALOpsDockerRemove@1
            displayName: 'ALOPS - Remove Docker Container'
            enabled: true

image
image image

An alterantive solution is to open the testresult.xml and make a hard fail if it consists substring failed="1"

waldo1001 commented 2 years ago

We are looking into this - we don't seem to see this behavior.
You example is also BC14, but you claim it also happens on newer versions?

NAVFreak commented 2 years ago

🤔 When refering to latest version I meant docker image.

Starting: ALOPS - Create Docker Image
==============================================================================
Task         : ALOps Docker Create
Description  : Create Docker image from Business Central Artifacts
Version      : 1.449.3024
Author       : Hodor
Help         : Create Docker image based on NAV/BC Artifacts.
==============================================================================
*** Validate configuration
*** Task Inputs:

name                      value                                                                                      
----                      -----                                                                                      
artifactspecification     Specific                                                                                   
artifactversion                                                                                                      
artifacttype              Sandbox                                                                                    
artifactcountry           se                                                                                         
versionselect             latest    
...
*** Artifact Url: [https://bcartifacts.azureedge.net/sandbox/20.2.41144.42006/se]
*** Build new Image based on [https://bcartifacts.azureedge.net/sandbox/20.2.41144.42006/se]
*** Artifact Type   : [sandbox]
*** Artifact Version: [20.2.41144.42006]
*** Artifact Country: [se]
*** OS: [2019]
*** OS version: [10.0.17763.2928dev]

Is nav_serverinstance a parameter to set BC version? I always thought this was nothing to bother about and was just a fictive name.

waldo1001 commented 2 years ago

Sorry, I misread. From the moment you use "UseDocker:true", it will take all settings from the container that it used in that pipeline. The rest is just default values which are unused.

We're looking into this .. weird thing is that we can't seem to repro this yet.. .

waldo1001 commented 2 years ago

@NAVFreak ,

From the moment you see this in your pipeline image We print that message at the same time we raise a VSTS-Error.

So the step in your output, does see ok.

But this continueOnError: true # <------------------- This setting decides if the build should fail if the tests fail is the wrong assumption, I think.. . ContinueOnError will .. well .. continue on error - so set this to false, and I think your issue is gone.

waldo1001 commented 2 years ago

@Solmyr1982, we found the source of your error.

The problem is as expected: a bug in BC14 where the LAST step was not documented in the results.xml.

Notice the output

*** Run Available Tests for suite [DEFAULT] with page [90409]
*** Connect TestClient with page [90409]
Show form {00016129-0000-0000-0108-0000836BD2D2}
Title: MV Test Tool Page Edit - Test Tool - DEFAULT
  Codeunit 54775 Geographic API TestSuite 1 IMV Success (6.852 seconds)
  Codeunit 54810 LocalAddressChgTestSuiteIMV Success (2.868 seconds)
  Codeunit 54812 Nodes API TestSuite IMV Success (2.094 seconds)
  Codeunit 54833 Shipping Aggr. TestSuite IMV Success (9.606 seconds)
  Codeunit 54853 Fullfil. Ord. Test Suite 1 IMV Success (4.269 seconds)
  Codeunit 54892 Ecom API TestSuite1 IMV Success (8.955 seconds)
  Codeunit 54986 POS Development Testsuite IMV Failure (9.15 seconds)

The last line fails, but is not in the result-file:

<?xml version="1.0" encoding="UTF-8"?>
<assemblies>
  <assembly name="54775 Geographic API TestSuite 1 IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:14" total="2" passed="2" failed="0" skipped="0" time="6.223">
    <collection name="Geographic API TestSuite 1 IMV" total="2" passed="2" failed="0" skipped="0" time="6.223" Skipped="0">
      <test name="Geographic API TestSuite 1 IMV:TestPostCodeUsed" method="TestPostCodeUsed" time="5.927" result="Pass" />
      <test name="Geographic API TestSuite 1 IMV:TestGeoLocationsImport" method="TestGeoLocationsImport" time="0.296" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54810 LocalAddressChgTestSuiteIMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:21" total="9" passed="9" failed="0" skipped="0" time="2.389">
    <collection name="LocalAddressChgTestSuiteIMV" total="9" passed="9" failed="0" skipped="0" time="2.389" Skipped="0">
      <test name="LocalAddressChgTestSuiteIMV:OnRun" method="OnRun" time="0" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54792" method="OpenReport54792" time="1.733" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54793" method="OpenReport54793" time="0.047" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54794" method="OpenReport54794" time="0.063" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54795" method="OpenReport54795" time="0.093" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54796" method="OpenReport54796" time="0.063" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54797" method="OpenReport54797" time="0.233" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54798" method="OpenReport54798" time="0.08" result="Pass" />
      <test name="LocalAddressChgTestSuiteIMV:OpenReport54799" method="OpenReport54799" time="0.077" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54812 Nodes API TestSuite IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:24" total="2" passed="2" failed="0" skipped="0" time="1.66">
    <collection name="Nodes API TestSuite IMV" total="2" passed="2" failed="0" skipped="0" time="1.66" Skipped="0">
      <test name="Nodes API TestSuite IMV:TestCollectionPointsImport" method="TestCollectionPointsImport" time="1.613" result="Pass" />
      <test name="Nodes API TestSuite IMV:UpdateShipToCityFromSellToCity" method="UpdateShipToCityFromSellToCity" time="0.047" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54833 Shipping Aggr. TestSuite IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:26" total="9" passed="9" failed="0" skipped="0" time="9.015">
    <collection name="Shipping Aggr. TestSuite IMV" total="9" passed="9" failed="0" skipped="0" time="9.015" Skipped="0">
      <test name="Shipping Aggr. TestSuite IMV:TestFieldsValidation" method="TestFieldsValidation" time="1.697" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:TestServiceOrderItemsServiceOrderNoValidation" method="TestServiceOrderItemsServiceOrderNoValidation" time="1.063" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:TestServiceTimeSlotsReservationAPI" method="TestServiceTimeSlotsReservationAPI" time="0.236" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:TestIKEASalesOrderPaymentDueDateTime" method="TestIKEASalesOrderPaymentDueDateTime" time="0.204" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:TestServiceOrderDisablePickingService" method="TestServiceOrderDisablePickingService" time="3.063" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:OpenPage54874Wizard" method="OpenPage54874Wizard" time="2.346" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:IKEAAfterSalesOrderWithItemsOpenCLServiceWizard" method="IKEAAfterSalesOrderWithItemsOpenCLServiceWizard" time="0.236" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:IKEAEmptySalesOrderOpenCLServiceWizard" method="IKEAEmptySalesOrderOpenCLServiceWizard" time="0.03" result="Pass" />
      <test name="Shipping Aggr. TestSuite IMV:IKEASalesOrderWithItemsOpenCLServiceWizard" method="IKEASalesOrderWithItemsOpenCLServiceWizard" time="0.14" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54853 Fullfil. Ord. Test Suite 1 IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:36" total="2" passed="2" failed="0" skipped="0" time="3.783">
    <collection name="Fullfil. Ord. Test Suite 1 IMV" total="2" passed="2" failed="0" skipped="0" time="3.783" Skipped="0">
      <test name="Fullfil. Ord. Test Suite 1 IMV:TestNewFields" method="TestNewFields" time="1.94" result="Pass" />
      <test name="Fullfil. Ord. Test Suite 1 IMV:TestFieldsEditability" method="TestFieldsEditability" time="1.843" result="Pass" />
    </collection>
  </assembly>
  <assembly name="54892 Ecom API TestSuite1 IMV" test-framework="PS Test Runner" run-date="2022-05-11" run-time="14:09:40" total="6" passed="6" failed="0" skipped="0" time="8.41">
    <collection name="Ecom API TestSuite1 IMV" total="6" passed="6" failed="0" skipped="0" time="8.41" Skipped="0">
      <test name="Ecom API TestSuite1 IMV:TestNewFields" method="TestNewFields" time="6.483" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:CreateEcomOrderFromAPIBuffer" method="CreateEcomOrderFromAPIBuffer" time="1.047" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:CreateEcomOrderFromAPIBufferWithNewContact" method="CreateEcomOrderFromAPIBufferWithNewContact" time="0.486" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:RemovePaymentDateTimeOnCancelOrder" method="RemovePaymentDateTimeOnCancelOrder" time="0.284" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:CheckPaymentOption" method="CheckPaymentOption" time="0.093" result="Pass" />
      <test name="Ecom API TestSuite1 IMV:AutomaticallyCreateEcomStatement" method="AutomaticallyCreateEcomStatement" time="0.017" result="Pass" />
    </collection>
  </assembly>
</assemblies>

We can only check the result-file to figure out if we should fail the pipeline. But since there is no error, there is nothing to fail.

We remember this was an issue in BC14 - but I couldn't find any documentation on that anymore :(. So the only thin I've got is "memory" .. .

Solmyr1982 commented 2 years ago

Hi @waldo1001 , thank you for your findings! Can you advise any solution or workaround to this issue? Maybe we can add one dummy test codeunit which will always be executed at the end, so if any of the previous fails, the pipeline will fail too?

waldo1001 commented 2 years ago

That would be a solution, imho. Give it the last number in your numberrange, and you should be good to go.. .

NAVFreak commented 2 years ago

Ahh... I tested it before I added stages but forgot to retest it once I had added stages. 🤦‍♂️ Thanks!

But this continueOnError: true # <------------------- This setting decides if the build should fail if the tests fail is the wrong assumption, I think.. . ContinueOnError will .. well .. continue on error - so set this to false, and I think your issue is gone.