search

fsprojects / TickSpec

Lean .NET BDD framework with powerful F# integration
Apache License 2.0
133 stars 23 forks source link

Big scenarios execution time #45

Closed AndrewRybka closed 5 months ago

AndrewRybka commented 3 years ago

Hi,

It takes ~111 ms to run all the 3 scenarios from the ByFramework/xUnit example when I run it as a standalone project:

image

But when I move that example to one of my test projects with 600+ other tests, the execution time raises to 5+ seconds, and each scenario is executed for more than a second:

image

Is there a way to reduce scenario execution time in the second case?

Thanks

mchaloupka commented 3 years ago

Hi,

are you able to share a project where it happens so we can reproduce it?

Potentially, are you able to run a profiler to see in which method it is spending most time?

Thanks, Milos

mchaloupka commented 3 years ago

Hi @AndrewRybka,

are you able to run a profiler to see what is the issue?

Thanks, Milos

AndrewRybka commented 2 years ago

Hi Milos,

Sorry for the delay with the answer.

Unfortunately, I can't share the initial project.

But I generated a project with ~2K unit test with a single assert in each (XUnit facts, theories, and FsCheck properties) in an attempt to replicate the issue. With that project it takes 900-1000 ms to execute scenarios from the ByFramework/xUnit example. With each scenario executed for 200-400 ms. And when I add additional scenarios, the overall execution time grows proportionally. So it seems that the number of tests in the project affects the execution time of each scenario. Here is the project with 2K tests: ByFramework.XUnit.zip

Also, I profiled the initial project while running only tests from the ByFramework/xUnit example. Here is the results overview: snapshot.txt

I can send the whole profiling snapshot, created by dotTrace, via email.

Thanks

plainionist commented 5 months ago

i observe similar behavior: the more tests (bigger scenarios) i add the slow the execution gets. any insights meanwhile what the root cause could be and how to improve?

mchaloupka commented 5 months ago

I have created a pull request with improvements: https://github.com/fsprojects/TickSpec/pull/69. That should improve it significantly.

@AndrewRybka thanks for providing the example project, although it took me forever to get to it, I have used it to figure out what is the problem.

plainionist commented 5 months ago

The improvement in execution time is HUGE in my case - thx a lot!

mchaloupka commented 5 months ago

@plainionist Out of curiosity, can you share what was roughly the improvement you have seen and how much time it still takes?