microsoft / testfx

MSTest framework and adapter
MIT License
738 stars 255 forks source link

Platform should expose a timeout option #3500

Closed thomhurst closed 2 months ago

thomhurst commented 2 months ago

I've seen in many test suites something go wrong before where something is hanging and the test suite just becomes stuck.

Adding a timeout option, that if exceeded, will send a cancellation signal to the framework's cancellation token could help give developers faster feedback and also save them money if they're using less compute time on their build agents.

The format could be a C# stringified timespan or you could come up with your own format.

Example:

dotnet run --timeout 00:10:00

AB#2200943

Evangelink commented 2 months ago

It seems like the responsibility of the test framework.

thomhurst commented 2 months ago

I just thought since there's a cancellation token passed to us in the request context, that'd be a good place to cancel it.

But no worries, it should be easy enough to implement in my framework.

Feel free to close :)

Evangelink commented 2 months ago

Reading again the ticket, I am thinking that maybe I didn't get what you were suggesting. Is your timeout a time per test or a global test execution timeout?

thomhurst commented 2 months ago

Sorry if it wasn't clear. A global timeout for the whole test session

Evangelink commented 2 months ago

Ok in that case I think it makes sense :) I was thinking about the platform handling a test method timeout and I was thinking that while technically we could do something along this line it seemed like something a lot easier to handle at framework level.

@MarcoRossignoli this seems like a useful request, WDYT?

Evangelink commented 2 months ago

Let's go with --timeout <val> where <val> will be either an int corresponding to timout in seconds or a string in the format <value>[h|m|s]