search

getgauge / gauge

Light weight cross-platform test automation
https://gauge.org
Apache License 2.0
2.99k stars 345 forks source link

Table driven scenarios do not follow scenario order and are always executed last #1960

Open hbenaicha opened 3 years ago

hbenaicha commented 3 years ago

Describe the bug Given a Spec that has 2 scenario and one table driven scenario, the table driven scenario will trigger last even if the order is scenario then table scenario then scenario

To Reproduce Steps (or project) to reproduce the behavior: image

Implementation: image

You would expect that execution is as follow :

  1. run scenario 1
  2. run table spec
  3. run scenario 2

However when you execute, scenario table is executed last, not 2nd

image

Gauge version used is 1.1.7

sriv commented 3 years ago

Gauge does not promise the order of scenario execution in a spec file. You can control the order of spec execution within a suite by using the --sort flag, but within a spec file gauge does not explicitly give a sort order.

hbenaicha commented 3 years ago

Does that mean that even for a file without any scenario table, the order of execution of the scenario is not known / defined ? That feels like a big problem that forces us to create a scenario per file if we want to have a sequence of execution in a specific order

sergueis commented 3 years ago

Gauge does not promise the order of scenario execution in a spec file. You can control the order of spec execution within a suite by using the --sort flag, but within a spec file gauge does not explicitly give a sort order.

Then what about this? "The scenarios should execute in the order they are specified when running in serial mode." - it's from Gauge FAQ ( https://docs.gauge.org/faqs.html?os=linux&language=python&ide=vscode#what-is-the-order-of-execution-of-scenarios-in-a-specification )

Gauge took a lot of good features from Cucumber, but didn't take sequential execution order of scenarios which is a must have.

sriv commented 3 years ago

@hbenaicha wrote:

Does that mean that even for a file without any scenario table, the order of execution of the scenario is not known / defined ?

Yes. And here's why:

Note that you can force gauge to follow an order by specifying the scenarios explicitly (ex: gauge run specs/example.spec:10 specs/example.spec:5 would run scenario in line 10 before line 5). But by default gauge run does not make any promises in this regard.

@sergueis wrote:

Then what about this?

Exactly what you quoted. Gauge will honour the order you specify as arguments to gauge run. See my example above.

If you passes an entire spec file or a directory, gauge will choose to decide the order and while you may see a sequence anecdotally, gauge does not guarantee that. i.e. you can use it at your own risk, the implementation is subject to change.

Gauge took a lot of good features from Cucumber

Incorrect. Gauge took inspiration from Twist, i.e. both gauge and twist are built by the same team. Both Twist and Cucumber started around the same time as independent initiatives.

but didn't take sequential execution order of scenarios which is a must have.

Gauge does not believe in this. The reasons are stated above.

sergueis commented 3 years ago

@sriv wrote:

Note that you can force gauge to follow an order by specifying the scenarios explicitly (ex: gauge run specs/example.spec:10 specs/example.spec:5 would run scenario in line 10 before line 5). But by default gauge run does not make any promises in this regard.

No, it doesn't work with the example above - scenario with table is still executed last.

Then how do I write a test when I need to do some preparation steps (in my case it would be a 1st separate scenario), then insert 20 records in a DB table (2nd scenario with table), and then do some processing/checking on these records (3rd scenario)? Again, explicit scenario order doesn't work in this case (ex: gauge run specs/example.spec:10 specs/example.spec:5).

zabil commented 3 years ago

Then how do I write a test when I need to do some preparation steps (in my case it would be a 1st separate scenario), then insert 20 records in a DB table (2nd scenario with table), and then do some processing/checking on these records (3rd scenario)?

In short, it's not possible.

In Gauge, scenarios must not depend on each other. This is because

sergueis commented 3 years ago

@zabil It's a pity that modern Gauge can't handle this very common use case that is easily handled by 13+ years old Cucumber.

Moreover: 1) If scenarios are independent units, why are they needed at all? One spec-file - one unit. And dir hierarchy for logical/functional grouping. 2) It's totally up to me - whether to run particular spec-file in parallel or not, to run particular tags or not, to run selected scenarios or not. They are all not mandatory requirements.

zabil commented 3 years ago

Gauge can't handle this very common use case that is easily handled by 13+ years old Cucumber.

I've noticed Cucumber re-appearing in comments despite an assertion earlier that Gauge and Cucumber are two different tools, that evolved very differently. So appreciate if we keep Cucumber out of the comments.

It's totally up to me - whether to run particular spec-file in parallel or not, to run particular tags or not, to run selected scenarios or not. They are all not mandatory requirements.

Gauge is an opinionated testing tool. To Gauge "It's more important to have a testable system than test cases". From our experience we've seen in the past that having contraints like this forces better design of test cases and the system under tests.

For that reason ordered execution of scenarios will not be supported. Hope that makes sense. I am closing this issue for now.

sriv commented 3 years ago

@sergueis - the use case is common, just that Gauge does not expect them to be in different scenarios as you have described.

Perhaps your notion of a Scenario does not match Gauge's.

Depending on whether you choose to make the DB inserts visible or transparent to the end users you can use Context/Teardown steps in the specification file, or execution hooks in the implementation.

sriv commented 3 years ago

@zabil - I propose that we leave this issue open, because it seems that Gauge does not honour the explicit order of scenarios passed to the gauge run command when there is a table driven scenario.

Also worth noting that Table Driven Scenarios is an experimental feature.

sergueis commented 3 years ago

@zabil Regarding Cucmber, why can't I compare Gauge with any other testing framework, regardless of their evolution/goals/contributors/age/etc?

If ordered (by default) execution of scenarios is not supported, please change this FAQ section "The scenarios should execute in the order they are specified when running in serial mode." to something like "Gauge can execute scenarios in any order" - just to not mislead users.

@sriv Thank you, I know about context/teardown but they are implicitly executed before/after each scenario which is not acceptable in our case (we have more that 3 scenarios in 1 spec-file and they are supposed to be executed sequentially). We have to get rid of table-driven scenario as a workaround.

hbenaicha commented 3 years ago

@sriv thanks for putting this as a bug, I agree and understand what you said below, but these are still cases where users wants to run in //, wants to run only failed cases etc...the general use case and the expectation is that Gauge would be following the order of execution of scenario / table driven one.

From that perspective it is a bug, so appreciate your support with this

@sriv wrote:

  • the scenarios being executed can change based on various parameters - the tags being applied, the table variables being used (or not), the explicit order being specified as part of gauge run
  • gauge recommends that the scenarios be independent. This is to avoid temporal coupling between scenarios which can be a pain to discover
  • note that if scenarios were dependent, and the 3rd scenario in the dependency failed, it would be pointless to run gauge run --failed because it will never get the dependencies included in the run.

Note that you can force gauge to follow an order by specifying the scenarios explicitly (ex: gauge run specs/example.spec:10 specs/example.spec:5 would run scenario in line 10 before line 5). But by default gauge run does not make any promises in this regard.

hbenaicha commented 3 years ago

note that this part though

Note that you can force gauge to follow an order by specifying the scenarios explicitly (ex: gauge run specs/example.spec:10 specs/example.spec:5 would run scenario in line 10 before line 5). But by default gauge run does not make any promises in this regard.

was tested in the above example and didn't seem to work for the table scenario

kga676 commented 3 years ago

Got same issue and spent 2 days for debugging, I used DataStore Hasmap

  1. generate value
  2. Use in step And I got issue when with table-driven spec step 2 has not able to find a value

Thanks for thiss issue, It saved me from madness

kga676 commented 3 years ago

btw witout table driven

spec1

scen1

scen2

scen3

still execute them in right order (1-2-3) (please DON'T change this behavior)

sergueis commented 3 years ago

Got same issue and spent 2 days for debugging, I used DataStore Hasmap

  1. generate value
  2. Use in step And I got issue when with table-driven spec step 2 has not able to find a value

Thanks for thiss issue, It saved me from madness

Unfortunately, one of the gauge authors has strong (and wrong) vision that scenaros in one spec-file should be totally independent, so... don't expect this bug to be fixed soon :)

zabil commented 3 years ago

@sergueis thanks for the feedback. Happy to be proven wrong! It will be interesting to see a clean and well designed implementation (perhaps a PR or a proof of concept) of making scenarios dependent on each other without the limitations discussed above.

sergueis commented 3 years ago

Don't make them dependent, just execute in order they are defined. It's that simple.

On Wed, 27 Oct 2021, 02:27 Zabil Cheriya Maliackal, < @.***> wrote:

@sergueis https://github.com/sergueis thanks for the feedback. Happy to be proven wrong! It will be interesting to see a clean and well designed implementation (perhaps a PR or a proof of concept) of making scenarios dependent on each other without the limitations discussed above.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/getgauge/gauge/issues/1960#issuecomment-952199871, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMBCAJSF7XMZXVC3RP62TWTUI36IXANCNFSM427F4CKA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

bpierce01 commented 1 year ago

Not sure if this issue is still being tracked, but I would also like to see scenarios execute in the order they are defined inside the spec file.

Like how @kga676 points out, scenarios are executed in order when there is no table driven scenario. But when there is a table driven scenario, then the execution order changes. It is inconsistent implementation. I think it's very reasonable to enforce that inside a spec file, the scenarios are always executed in the order defined.