search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.5k stars 2.61k forks source link

Run Functional Tests: The test run was aborted, failing the task. #6738

Closed yorhpokie closed 6 years ago

yorhpokie commented 6 years ago

Environment

Issue Description

Run Functional Tests task fails periodically with "The test run was aborted, failing the task. The DTAExecutionHost.exe.log shows this exception:

E, 4440, 16, 2018/03/19, 14:34:14.104, TV3-AGNT-0909\DTAExecutionHost, TestExecutionServiceCommandQueueListener.SendStopCommandToHandleException : An exception occured in queue listener. System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.VisualStudio.TestService.AgentExecutionHost.TestExecutionServiceCommandQueueListener.d__7.MoveNext() Sending Stop command...

Error logs

2018-03-19T21:34:00.5677513Z DistributedTests: Test run '4630725' is in 'InProgress' state. 2018-03-19T21:34:00.5677513Z DistributedTests: Total Tests : 586, Passed Tests : 513 2018-03-19T21:34:10.6808466Z DistributedTests: Test run '4630725' is in 'Aborted' state. 2018-03-19T21:34:10.6808466Z DistributedTests: Total Tests : 586, Passed Tests : 513 2018-03-19T21:34:20.6843066Z ##[warning]DistributedTests: Test run is aborted. Logging details of the run logs. 2018-03-19T21:34:20.6843066Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run is aborted. Logging details of the run logs. 2018-03-19T21:34:20.7155519Z ##[warning]DistributedTests: New test run created. 2018-03-19T21:34:20.7155519Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: New test run created. 2018-03-19T21:34:20.7155519Z Test Run queued for Project Collection Build Service (TEAM FOUNDATION). 2018-03-19T21:34:20.7155519Z 2018-03-19T21:34:20.7155519Z ##[warning]DistributedTests: Test execution started. Test run id : 4630725 2018-03-19T21:34:20.7155519Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test execution started. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run timed out. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run timed out. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run aborted. Test run id: 4630725 2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run aborted. Test run id: 4630725 2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run timed out. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run timed out. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run timed out. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run timed out. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run timed out. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run timed out. Test run id : 4630725 2018-03-19T21:34:20.7311857Z ##[error]System.Exception: The test run was aborted, failing the task. 2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=error;]System.Exception: The test run was aborted, failing the task. 2018-03-19T21:34:20.7780676Z ##[debug]DistributedTests: Deleted test settings with id : 1052771 2018-03-19T21:34:20.8718067Z ##[error]PowerShell script completed with 1 errors. 2018-03-19T21:34:20.8718067Z ##[section]Finishing: Run Tests

DTAExecutionHost.exe.log

ShreyasRmsft commented 6 years ago

@yorhpokie This looks like a known issue we had in DTA RFT. As of now in the latest Tfs (2018 Update 2) the DTA RFT task has been deprecated. We are asking folks to move to the VsTest v2 task that is capable of parallel execution both in release and build (But in Tfs 2017 update 3 only parallel execution releases is supported). Are you guys running tests in Build or Release?

akzhn commented 6 years ago

It always happens when RunTests task runs for 6 hours.

Setting up 'TestSessionTimeout' in .runsettings not working, while .testsetting (with 'runTimeout' param inside) is not supported by MsTestV2.

ShreyasRmsft commented 6 years ago

@nshr12 The timeout is not for MsTestV2 but for the RFT task. You should be able to specify runTimeout with .testsettings and it should override the default 6 hour timeout of the RFT task.

Just override your preferred timeout value in the testsettings below

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="ci" id="adc5287b-c979-4581-b076-044af81d138b" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a ci run.</Description>
  <Deployment enabled="false" />
  <Execution>
    <Timeouts runTimeout="36000000" testTimeout="0" />
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution applicationBaseDirectory="c:\tests" />
      </UnitTestRunConfig>
    </TestTypeSpecific>
    <AgentRule name="LocalMachineDefaultRole">
    </AgentRule>
  </Execution>
  <Properties />
</TestSettings>
akzhn commented 6 years ago

@ShreyasRmsft , thanks, but how can I use both .runsettings and .testsettings at the same time?

As I sad above, .testsettings is not supported by mstestv2..

yorhpokie commented 6 years ago

Thank you for the reply, @ShreyasRmsft! We're running tests in Build. I guess we'll just retry the tests until we upgrade to TFS 2018.

ShreyasRmsft commented 6 years ago

@nshr12 Sadly there's no way to use both test settings and run settings together. In VSTS we're moving to replace testsettings with runsettings and everything that was supported with test settings will be supported by runsettings.

ShreyasRmsft commented 6 years ago

@yorhpokie Was your issue also with 6 hour timeouts or arbitrary duration timeouts? If they were 6 hour timeouts you can add more agents to reduce the overall execution time to less than 6 hours or divide the tests into 2 halves and run them via 2 instances of the tasks. So that each run happens within 6 hours as a workaround. @nshr12 you can do this too.

yorhpokie commented 6 years ago

@ShreyasRmsft: our tests were aborted after 30 or 40 minutes. It was definitely not 6 hours.

ShreyasRmsft commented 6 years ago

@yorhpokie the arbitrary timeout is known issue I was talking about. We fixed it in 2018 Update 1. The 6 hour timeout was configurable always just that runsettings support was added later.