search

microsoft / navcontainerhelper

Official Microsoft repository for BcContainerHelper, a PowerShell module, which makes it easier to work with Business Central Containers on Docker.
MIT License
385 stars 246 forks source link

Executing Tests in Parallel #932

Closed marknitek closed 4 years ago

marknitek commented 4 years ago

To improve automated tests execution time in the builds i am experimenting with different common techniques. One of them was to execute the tests in parallel by creating copies of the cronus company beforehand and then splitting the codeunits in a test suite and executing them on the different companies in parallel by using powershell jobs.

This would potentially enable a much faster build execution when having multiple tests.

Yes you can offload long running test suites to nighly builds but as we all know this is just a workaround and does not really get along with ci/cd principles

When i do this i get an internal server error (500) and other errors for any job but the first one. So i guess the implementation that executes the tests in Run-Test does not support concurrent sessions at the moment?
Since i can execute the test manually by running multiple sessions in the webclient i think it should be doable.
Can the Run-Test function be changed so support this scenario?

Here the code i'am using for testing this (for simplicity i just want to execute the same test codeunit in all 5 companies)

$Jobs = @()
Write-Host "Running Tests in Parallel (Parallel: $companyCount, Codeunits per Batch: $testCountPerSet)"
$companyCount = 5
for ($i = 0; $i -lt $companyCount; $i++) {
    $companyName = "Test$i"   
    $CodeunitFilter = "70001"

    # Run Tests in Parallel
    Write-Host Running Test Coduenits $($subset.Name -join ", ") in Company $companyName        

    $Jobs += Start-Job { param($containerName, $credential, $testResultsFile, $TestSuite, $CodeunitFilter, $companyName)            
        Run-TestsInNavContainer $containerName -companyName $companyName -credential $credential -testSuite $TestSuite -testCodeunit $CodeunitFilter -AppendToXUnitResultFile -XUnitResultFileName $testResultsFile
    } -ArgumentList $containerName, $credential, $testResultsFile, "DEFAULT", $CodeunitFilter, $companyName 

}    
$Jobs | Wait-Job
$Jobs | Receive-Job

Errors

CommunicationError : Error while copying content to a stream.
at <ScriptBlock>, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 89
at InvokeInteractionAndCatchForm, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 230
at OpenForm, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 241
at Run-Tests, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\PsTestFunctions.ps1: line 267
at <ScriptBlock>, <No file>: line 53
Current Interaction: Microsoft.Dynamics.Framework.UI.Client.Interactions.OpenFormInteraction
Time spend: 0 seconds
Cannot open page 130455. You might need to import the test toolkit to the container and/or remove the folder C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5 and retry. You might also have URL or Company name wrong.
at Run-Tests, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\PsTestFunctions.ps1: line 269
at <ScriptBlock>, <No file>: line 53
WARNING: Caution: Your program license expires in 8 days.
  Codeunit 70001 LOGTESearch And Replace Success (2.232 seconds)
CommunicationError : Error while copying content to a stream.
at <ScriptBlock>, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 89
at InvokeInteractionAndCatchForm, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 230
at OpenForm, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 241
at Run-Tests, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\PsTestFunctions.ps1: line 267
at <ScriptBlock>, <No file>: line 53
Current Interaction: Microsoft.Dynamics.Framework.UI.Client.Interactions.OpenFormInteraction
Time spend: 0 seconds
Cannot open page 130455. You might need to import the test toolkit to the container and/or remove the folder C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5 and retry. You might also have URL or Company name wrong.
at Run-Tests, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\PsTestFunctions.ps1: line 269
at <ScriptBlock>, <No file>: line 53
CommunicationError : Response status code does not indicate success: 500 (Internal Server Error).
at <ScriptBlock>, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 89
at AwaitState, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 194
at OpenSession, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 172
at Initialize, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 59
at ClientContext, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 26
at New-ClientContext, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\PsTestFunctions.ps1: line 40
at <ScriptBlock>, <No file>: line 51
ClientSession State is Uninitialized (Wait time 10 seconds)
at AwaitState, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 206
at OpenSession, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 172
at Initialize, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 59
at ClientContext, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 26
at New-ClientContext, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\PsTestFunctions.ps1: line 40
at <ScriptBlock>, <No file>: line 51
CommunicationError : Response status code does not indicate success: 500 (Internal Server Error).
at <ScriptBlock>, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 89
at AwaitState, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 194
at OpenSession, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 172
at Initialize, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 59
at ClientContext, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 26
at New-ClientContext, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\PsTestFunctions.ps1: line 40
at <ScriptBlock>, <No file>: line 51
ClientSession State is Uninitialized (Wait time 10 seconds)
at AwaitState, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 206
at OpenSession, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 172
at Initialize, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 59
at ClientContext, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\ClientContext.ps1: line 26
at New-ClientContext, C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5\PsTestFunctions.ps1: line 40
at <ScriptBlock>, <No file>: line 51
Cannot open page 130455. You might need to import the test toolkit to the container and/or remove the folder C:\ProgramData\NavContainerHelper\Extensions\lep15-ext-dev\PsTestTool-5 and retry. You might also have URL or Company name wrong.
In C:\Users\Marco\source\repos\Logico-Powershell\Modules\Logico-AL\Tasks\Run-LocalTest.ps1:81 Zeichen:5

Eventlog

Microsoft.AspNetCore.Server.Kestrel
Connection id "0HLUOCDJGKKFP", Request id "0HLUOCDJGKKFP:00000001": An unhandled exception was thrown by the application.

System.ArgumentNullException: Value cannot be null.
Parameter name: performanceCounterFactory
   at Microsoft.Dynamics.Framework.UI.WebBase.Diagnostics.InteractionDiagnosticsRecorder..ctor(DiagnosticsSettings diagnosticsSettings, IInteractionNotifier notifier, IPerformanceCounterFactory performanceCounterFactory, Func`1 createStopwatch) in s:\repo\src\platform\client\web\Prod.Client.WebBase\Diagnostics\InteractionDiagnosticsRecorder.cs:line 45
   at Microsoft.Dynamics.Framework.UI.WebBase.ClientSessionBase.InitCore() in s:\repo\src\platform\client\web\Prod.Client.WebBase\Session\ClientSessionBase.cs:line 467
   at Microsoft.Dynamics.Nav.Client.ClientService.NavClientServiceSession.InitCore() in s:\repo\src\platform\client\ClientService\Prod.Client.ClientService\Session\NavClientServiceSession.cs:line 131
   at Microsoft.Dynamics.Framework.UI.ClientSessionCore.Init() in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Session\ClientSessionCore.cs:line 417
   at Microsoft.Dynamics.Framework.UI.WebBase.ClientSessionManagerBase.CreateClientSessionCore(RequestContext requestContext, String sessionId) in s:\repo\src\platform\client\web\Prod.Client.WebBase\Session\ClientSessionManagerBase.cs:line 714
   at Microsoft.Dynamics.Framework.UI.ClientService.ClientServiceSessionManager.CreateClientSessionCore(RequestContext requestContext, String sessionId) in s:\repo\src\platform\client\ClientService\Prod.Client.ClientServiceFwk\Session\ClientServiceSessionManager.cs:line 54
   at Microsoft.Dynamics.Framework.UI.ClientSessionManagerCore.CreateClientSession(RequestContext requestContext, String sessionId) in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Session\ClientSessionManagerCore.cs:line 568
   at Microsoft.Dynamics.Framework.UI.ClientSessionManagerCore.EnsureSession(RequestContext requestContext, String sessionId) in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Session\ClientSessionManagerCore.cs:line 448
   at Microsoft.Dynamics.Framework.UI.WebBase.ClientSessionManagerBase.EnsureSession(RequestContext requestContext) in s:\repo\src\platform\client\web\Prod.Client.WebBase\Session\ClientSessionManagerBase.cs:line 529
   at Microsoft.Dynamics.Framework.UI.ClientService.ClientService.OpenSession(ClientSessionParameters parameters, RequestContext requestContext) in s:\repo\src\platform\client\ClientService\Prod.Client.ClientServiceFwk\Service\ClientService.cs:line 53
   at Microsoft.Dynamics.Nav.WebClient.Controllers.ClientServiceController.<OpenSession>d__6.MoveNext() in s:\repo\src\platform\client\web\Prod.Client.WebCoreApp\Controllers\Services\ClientServiceController.cs:line 96

and 

Microsoft.AspNetCore.Server.Kestrel
Connection id "0HLUOCDJGKKFP", Request id "0HLUOCDJGKKFP:00000002": An unhandled exception was thrown by the application.

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Dynamics.Framework.UI.LogicalForm.get_InboundCommunicationChannels() in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Forms\LogicalForm.cs:line 1155
   at Microsoft.Dynamics.Framework.UI.LogicalForm.get_PendingMessagesCount() in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Forms\LogicalForm.cs:line 1174
   at Microsoft.Dynamics.Framework.UI.Client.LogicalFormSerializer.WriteControl(LogicalControlSerializerContext context, IObjectWriter writer, LogicalForm logicalControl) in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Client\Serialization\LogicalControlSerializers\LogicalFormSerializer.cs:line 49
   at Microsoft.Dynamics.Framework.UI.Client.LogicalControlSerializer`2.WriteCore(ObjectSerializerContext context, IObjectWriter writer, LogicalControl value) in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Client\Serialization\LogicalControlSerializers\LogicalControlSerializer.cs:line 481
   at Microsoft.Dynamics.Framework.UI.Client.LogicalChangeSetSerializer.Write(LogicalControlSerializerContext context, IObjectWriter writer, UISessionDialogToShowEventRaisedChange eventChange, IObjectSerializerFactory`1 clientLogicalControlBuilderFactory) in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Client\Interactions\Changes\LogicalChangeSetSerializer.cs:line 477
   at Microsoft.Dynamics.Framework.UI.Client.LogicalChangeSetSerializer.Write(LogicalControlSerializerContext context, IObjectWriter writer, IEnumerable`1 changes, IObjectSerializerFactory`1 serializerFactory) in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Client\Interactions\Changes\LogicalChangeSetSerializer.cs:line 50
   at Microsoft.Dynamics.Framework.UI.Client.LogicalChangeSetSerializer.Write(LogicalControlSerializerContext context, IObjectWriter writer, LogicalChangeSet logicalControlSet, IObjectSerializerFactory`1 serializerFactory) in s:\repo\src\platform\client\Shared\Prod.ClientFwk\Client\Interactions\Changes\LogicalChangeSetSerializer.cs:line 36
   at Microsoft.Dynamics.Nav.WebClient.Controllers.ClientServiceController.SerializeInteractionResult(JsonObjectWriter objectWriter, ClientServiceSession session, IInteractionResult interactionResult) in s:\repo\src\platform\client\web\Prod.Client.WebCoreApp\Controllers\Services\ClientServiceController.cs:line 147
   at Microsoft.Dynamics.Nav.WebClient.Controllers.SerializeToJsonResult.ExecuteResultAsync(ActionContext context) in s:\repo\src\platform\client\web\Prod.Client.WebCoreApp\Controllers\Services\JsonRpc\SerializeToJsonResult.cs:line 46
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeResultAsync>d__20.MoveNext()
freddydk commented 4 years ago

I am not sure that the test framework supports this and every test also needs to be very carefully architected in a way that it doesn't encounter locks from other tests. If you have a lot of tests, you could spin up 2 or 3 containers and ask every container to execute some of them. Azure DevOps does (as far as I know) support to run multiple steps side by side and this solution would be safe.

marknitek commented 4 years ago

Well the testframework seem to support it since its working when executed manually. I understand that it might not be something that is easy to fix or of any priority.

But to be honest i expected a discussion about that topic since it is a important topic. We need to do tests and tests are slow in BC so i thought about ways to improve it. Another idea was to implement TIA (test impact analysis) to selectivly execute only affected tests.

Running multiple containers to achieve parallel testing is a resource waste in my opinion. BC already needs so much resources to run... I dont think that test architecture is a major problem in that scenario. When running in different companies table locks are nearly impossible and tests should not affect each other. Iam a bit disapointed about the lack of discussion but maybe this is the wrong plattform

freddydk commented 4 years ago

So, I can see that you are running every test in a different company, then there wouldn't be any data collision. I still think that the way the AL Test Tool is created, it doesn't take this scenario into account, but I must admit that I haven't tried. The AL Test Tool is a temporary solution and spending a lot of time in optimizing this is not a good use of my time, sorry. In the end, the modern dev. team will create a replacement, where tests can be executed through the dev. endpoint, code coverage can be achieved etc. etc. This will allow us to check which tests are covering the code we are changing and only run the tests that actually are necessary. It might also allow for parallel execution of tests. I do not have a date for this.