Support parallel execution of tests

[mfulton26]

There are various ways to implement. Personally I believe I prefer to be able to execute group and test blocks myself using my own ExecutorService, etc. but that doesn't work today as Spek "pushes" blocks onto a queue/tree and executes everything itself serially.

One of the big things I don't like about many test frameworks is that they try to provide their own DSL for parallel execution with limited configuration options, etc. instead of simply letting the test developer call the framework using their own parallel/concurrent code so that they can control when/where/how/etc.

JUnit 5 supports practically full dynamic trees of containers (groups) and tests. As such, Spek could potentially be changed to execute groups/tests live instead of "discovering" all tests first and then "executing" the tests separately. This would allow test developers to use for loops, if statements, executor services, etc. to execute their groups and tests as they please.

[artem-zinnatullin]

You already can use loops and flow control

On Fri, Aug 26, 2016 at 6:45 PM, Mark Fulton notifications@github.com wrote:

There are various ways to implement. Personally I believe I prefer to be able to execute group and test blocks myself using my own ExecutorService, etc. but that doesn't work today as Spek "pushes" blocks onto a queue/tree and executes everything itself serially.

One of the big things I don't like about many test frameworks is that they try to provide their own DSL for parallel execution with limited configuration options, etc. instead of simply letting the test developer call the framework using their own parallel/concurrent code so that they can control when/where/how/etc.

JUnit 5 supports practically full dynamic trees of containers (groups) and tests. As such, Spek could potentially be changed to execute groups/tests live instead of "discovering" all tests first and then "executing" the tests separately. This would allow test developers to use for loops, if statements, executor services, etc. to execute their groups and tests as they please.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JetBrains/spek/issues/108, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3DLG0SfIB915koNpWqEuqtRr-BYsks5qjwoCgaJpZM4JuNSf .

[mfulton26]

@artem-zinnatullin Yes but to a limited extent. e.g. You can not make a test dependent on results of a previous test or use results of a test to data-drive further tests, etc. As it is today (until Kotlin 1.1 comes out) you cannot do expensive set-up lazily either in a group. Discovering and executing groups and tests at the same time could be very advantageous to give developers further control.

[artem-zinnatullin]

Though I find making tests dependent on other tests, dynamic tree will great

On Fri, Aug 26, 2016 at 7:01 PM, Mark Fulton notifications@github.com wrote:

@artem-zinnatullin https://github.com/artem-zinnatullin Yes but to a limited extent. e.g. You can not make a test dependent on results of a previous test or use results of a test to data-drive further tests, etc. As it is today (until Kotlin 1.1 comes out) you cannot do expensive set-up lazily either in a group. Discovering and executing groups and tests at the same time could be very advantageous to give developers further control.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JetBrains/spek/issues/108#issuecomment-242776542, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3GGh0NK47JQV8CHbbZgfJJ3QmVzrks5qjw3VgaJpZM4JuNSf .

[raniejade]

AFAIK JUnit 5's dynamic tests doesn't support groups. ExecutorServices will just complicate things, since tests are created dynamically we need a reliable way to tell Spek that all tests are generated. You can already use any control flow and with coroutine support in Kotlin 1.1 we can do a lot more.

[artem-zinnatullin]

Oh no, wait, please do not tight Spek with Kotlin's coroutines

On Sat, Aug 27, 2016 at 2:42 PM, Ranie Jade Ramiso <notifications@github.com

wrote:

AFAIK JUnit 5's dynamic tests doesn't support groups. ExecutorServices will just complicate things, since tests are created dynamically we need a reliable way to tell Spek that all tests are generated. You can already use any control flow and with coroutine support in Kotlin 1.1 we can do a lot more.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JetBrains/spek/issues/108#issuecomment-242912464, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3Ek_A7mfHG8pwpfFddfbmg3CHWJNks5qkCKngaJpZM4JuNSf .

[raniejade]

@artem-zinnatullin Why is that?

[artem-zinnatullin]

Because as @mfulton26 pointed:

One of the big things I don't like about many test frameworks is that they try to provide their own DSL for parallel execution with limited configuration options, etc. instead of simply letting the test developer call the framework using their own parallel/concurrent code so that they can control when/where/how/etc.

It's better to not tight Spek to specific API for parallelization, for example I (we) would like to use RxJava for that, someone would like to use Executors API, others will use Reactor, etc, etc.

[raniejade]

Ahh, yep definitely. What I meant was you can do more complex test setup with coroutines compared to the control flows (which we currently have).

[artem-zinnatullin]

Ah, ok!

On 28 Aug 2016 6:47 am, "Ranie Jade Ramiso" notifications@github.com wrote:

Ahh, yep definitely. What I meant was you can do more complex test setup with coroutines compared to the control flows (which we currently have).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JetBrains/spek/issues/108#issuecomment-242954488, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3Iipy7UB7nlJ42z1lurDlHcfcXrYks5qkQTagaJpZM4JuNSf .

[raniejade]

One thing I can think via extensions that allow to tap into Spek's lifecycle. Been playing around with extending Spek, here's what I have so far https://github.com/JetBrains/spek/blob/master/spek-junit-platform-engine/src/test/kotlin/org/jetbrains/spek/engine/ExtensionTest.kt.

[nizienko]

Hello, I tried to execute tests in parallel with 'maxParallelForks 2' in gradle config, which is works well on simple JUnit5 project without spek. All tests were executed twice in 2 threads.

[corlaez]

This feature would be a nice addition! A button that allows parallel execution would be enough for my needs. Similar to this junit4 plugin: https://plugins.jetbrains.com/plugin/12959-junit4-parallel-runner

[corlaez]

https://kotlin.libhunt.com/spek-latest-version

Since 2.0.13 we have these gradle properties that allow parallel execution:

spek2.discovery.parallel.enabled - controls whether discovery is done in parallel or not, by default it is disabled. The presence of this property regardless of value will enable parallel discovery.

spek2.execution.parallel.enabled - controls whether execution is done in parallel or not, by default it is disabled. The presence of this property regardless of value will enable parallel execution.

A note for parallel execution ✅ Parallelism is at a class level not individual test scopes. Spek uses coroutines under the hood to execute tests in parallel, essentially spek2.execution.parallel.enabled just controls which CoroutineDispatcher is used. For parallel executions Dispatchers.Default is used, otherwise the one provided by runBlocking is used.

Would that be enough to close this issue?

[raniejade]

@corlaez When this ticket was created the plan is to have test scope level parallelism, not just class level. I might have to re-think this a bit further - the real question is: is class/spek level parallelism enough?

[corlaez]

got, got it. Yeah I have noticed that 2.0.13 will execute tests within a class in parallel but at a class level the execution is still sequential.

[corlaez]

What I am used to is the way that Spock running with JUni4 Parallel Tests (https://plugins.jetbrains.com/plugin/12959-junit4-parallel-runner) works: Spock would instantiate the fields every time it is about to run a test. That means that in parallel each test works with its own instances. It sounds like wasted objects and garbage collection triggering more often, and it probably does but it has no impact as we have way more RAM and GC times are negligible. The nice thing is that I don't have to clear state or worry about racing conditions.

I am not even sure that the class parallelism we have right now works like that (or if spek in general is implemented in such a way), I have to do some digging.

[LisPI]

Hello, I tried to execute tests in parallel with 'maxParallelForks 2' in gradle config, which is works well on simple JUnit5 project without spek. All tests were executed twice in 2 threads.

We're experiencing the same issue. Maybe there are any workaround? btw looks like 2.0.4 the first Spek version with the problem