Test execution sometimes hangs when running in Azure DevOps using dotnet test

[stevozilik]

Hi,

I've been experiencing NUnit tests getting stuck when running in Azure Devops yml pipeline using the dotnet test ... command

Background:

• NUnit 4.1.0 (same behavior with previous versions)
• NUnit3TestAdapter 4.6.0 (same behavior with previous versions)
• Microsoft.NET.Test.Sdk 17.11.0 (same behavior with previous versions)
• .Net 8
• Azure DevOps running on ubuntu-latest or windows-latest (no difference)
• SpecFlow.NUnit 3.9.74

Observed Behavior (test logs):

• Test execution sometimes stops after one of the tests (inconsistent) and never completes. This is the log: ... legitimate test logs ... 2024-08-29T13:55:41.6129069Z -> duration: ScenarioHooks.AfterScenario(): 1.0s 2024-08-29T13:55:41.7389440Z * Expected log line with the test outcome, observed in previous tests, consistently missing when the tests get stuck (missing) 2024-08-29T13:55:41.6858716Z Passed TestName [1 s] * [note the time gap below, cancellation triggered by the job.timeoutInMinutes] 2024-08-29T14:00:28.8388057Z ##[error]The Operation will be canceled. The next steps may not contain expected logs. 2024-08-29T14:00:28.8557105Z ##[error]The operation was canceled. 2024-08-29T14:00:28.8560745Z ##[section]Finishing: Running Tests

It looks like something gets stuck and I'm struggling to figure out where to even begin

similar dotnet test ... command testing the same assembly locally works consistently ok

I'd like to ask for help how to diagnose this to get to the bottom of why it gets stuck

I've tried adding --verbosity diag --logger "console;verbosity=detailed" -- "NUnit.Consoleout=1 NUnit.InternalTraceLevel=Debug NUnit.Verbosity=5" to the dotnet test ... command but nothing jumps out as helpful

[dragojs]

Started experiencing exactly this some time ago and couldn't find the problem either. Gonna try to dump the logs in a file next run and see what I find

[OsirisTerje]

Sorry for not commenting on this earlier.

This is very hard to give any helpful comment on, there is too little information to go on.

Some questions:

Does the issue remain if you reduce the code to a single test (or just a few)? See the Specflow is involved. Does the same happens without specflow? Does it work locally ? Have you enabled the dumps in the NUnit3TestAdapter (see config options for a runsettings file) Have you checked the different possible timeout values and are you within those limits? Can you produce a repro solution ? Enabling diagnostic logs as you mention is a very good idea.

[dragojs]

Thank you for commenting as early as you did. Very good questions all of them, I am going to try and come back with a result.