getgauge / gauge

Light weight cross-platform test automation
https://gauge.org
Apache License 2.0
2.99k stars 344 forks source link

SpecDatastore is not reset when multiple specs are executed on the same runner #2270

Open F7502 opened 2 years ago

F7502 commented 2 years ago

Describe the bug We are running multiple test specs (using the gauge-ts runner) in parallel and we have more test specs than runners. In this setup some test specs are executed on the same runner sequentially. We observed that the Specification DataStore is not properly reset between such sequential test spec executions although the documentation clearly states: Values are cleared after every specification is executed.

I digged a bit into the code and found this here: https://github.com/getgauge/gauge/blob/363eb96c28be886bb688a9dfc0d0b0987d4d4d08/execution/specExecutor.go#L84

If executeBefore is false, the spec data store will not be reset before the spec is executed. However, when multiple specs are run on the same runner, executeBefore is only true for the first spec: loop over all specs: https://github.com/getgauge/gauge/blob/363eb96c28be886bb688a9dfc0d0b0987d4d4d08/execution/simpleExecution.go#L129 setting to false after first spec: https://github.com/getgauge/gauge/blob/363eb96c28be886bb688a9dfc0d0b0987d4d4d08/execution/simpleExecution.go#L134

Expected behavior The specification data store should be reset before every test spec execution.

Versions: "@getgauge/cli": "1.4.3"

zabil commented 2 years ago

Thanks for pointing out the issue. Do you want to send a pull request to fix this issue as you've already figured the spot?

sriv commented 2 years ago

@barnesjohnraymond has opened #2299 to address this, but I worry that the proposed change will break table driven execution.

I am going to try and replicate this issue, @F7502 / @@barnesjohnraymond - do you have a sample project that can help me replicate this? If not, I'll try to create something.