search

AArnott / Xunit.StaFact

Run your xunit-based tests on an STA thread with the WPF Dispatcher, a WinForms SynchronizationContext, or even a cross-platform generic UI thread emulation with a SynchronizationContext that keeps code running on a "main thread" for that test.
Other
92 stars 31 forks source link

StaFact facts breaks the concurrency limitations of Xunit #53

Closed lg2de closed 3 years ago

lg2de commented 3 years ago

Xunit has an internally implemented limitation of tests to be executed concurrently. This is implemented based on fixed number of threads executing the tests.

All Fact implementations of StaFact packages creating its own thread, e.g. UIFact, will break the Xunit limitation because the Xunit thread is released after UIFact test execution has been started. I think this causes unexpected high CPU load while test execution resulting in accidental failed test execution, e.g. fluentassertions/fluentassertions#1391 and https://ci.appveyor.com/project/dennisdoomen/fluentassertions/builds/37160306

I've created and attached a sample project showing the problem. For demonstration I've configured the limitation to 2 concurrent tests. All standard Fact tests completing successfully. Most UIFact tests will fail because the maximum number of tests is exceeded.

StaFactTests.zip

aesalazar commented 3 years ago

We occasionally see our tests fails or even hang on our build servers which is mentioned in one of those threads. Have you seen this as well?

lg2de commented 3 years ago

Yes, they have been failed if they rely on timing (assertion of execution time). They did NOT hang, at least we haven't noticed that so far.

AArnott commented 3 years ago

Thanks for reporting. Any idea how we would fix this? I have no experience with the xunit parallel execution throttle. The xunit methods are mostly async, so it seems designed that tests not always block a thread, so if xunit's throttling assumes the test blocks a thread, that already seems broken without UIFact's help. Can you repro this problem with any [Fact]-based async Task returning method that yields right away with .ConfigureAwait(false)?

lg2de commented 3 years ago

I think it should be ok to block the xunit thread while UIFact thread (and all other thread in this extension) is executing. Synchronous thread will "block" the thread too. Let say, the UIFact thread replaces temporary the xunit thread.

Can you repro this problem with any [Fact]-based async Task returning method that yields right away with ConfigureAwait(false)?

I could not really understand your question. And, what is the background of this question?

AArnott commented 3 years ago

what is the background of this question?

I am trying to establish how xunit throttles test runs when tests are asynchronous. Because whatever that mechanism is, we should reuse it for UIFact.

lg2de commented 3 years ago

Xunit starts as many threads as tests should be in parallel. All testing work is distributed to these worker threads.

With the additional threads of StaFact the mechanism cannot be reused, I guess. Instead I think the the Xunit thread should be blocked while StaFact thread is running.

Maybe @xunit can help?

lg2de commented 3 years ago

@AArnott , when do you plan to create a new release with this issue fixed?

AArnott commented 3 years ago

I'll try to get it out later today.

AArnott commented 3 years ago

This is now available as https://github.com/AArnott/Xunit.StaFact/releases/tag/v1.1.5-alpha

lg2de commented 2 years ago

Hey @AArnott, could you please create a new public release? I have a new project which must not use alpha or beta releases.

lg2de commented 2 years ago

We found another issue... :( Please wait with the release.