Filtering by Line Number

[olyv]

Hi,

I am using CucumberSwift for UI automation. And of the weakness of UI automation is the tests stability. Sometimes it happens to have flickering test which I might want to re-run. It's not a problem to rerun single scenario but it becomes troublesome when there is a scenario outline with several examples and only one example is failed.

The desired solution exists in Cucumber-JVM but not in CucumberSwift. Cucumber-JVM existing solution is described in http://grasshopper.tech/557/ ("Filtering by Line Number" chapter). In terms of technical implementation, it seems like it is needed to add uri property to scenario entity and modify logic of scenario creation from scenario outline. And it sounds like quite an effort, to be honest.

Please let me know if I can provide with more details about this feature request. Thanks.

[Tyler-Keith-Thompson]

I'm normally pretty against re-inventing the wheel but line number seems like an awful way to identify a scenario to run given its potential to change.

I understand how being able to run one of several examples would be useful. What are your thoughts on breaking the mold? Perhaps allow tagging at a more granular level? Or use some other indicator of which example to rerun?

[olyv]

I have to admit, I was under impression that 'filtering by line' was tested by cucumber community and proved its usefulness. Apart from that it exists in Ruby, Java and JS ports of cucumber.

As for granular tagging -- possible but it will require additional juggling with tags and probably feature won't be that readable. Other possibilities? Hmmm, not sure to be honest

[Tyler-Keith-Thompson]

Very well the wheel shall stay very round and invented...with a twist!

So I brought shouldRunWith(tags:[String]) -> Bool in because people may have complex conditional needs. Only run with the tag "run" if it's a blood moon kind of thing. How would you feel about shouldRun(scenario:Scenario, withTags: [String]) -> Bool.

That not only gives access to the concrete (already parsed) object but also the line numbers. We'll add some kind of property on scenarios, features, and steps that gives you it's position (probably a line/column tuple).

Then it's 100% in the hands of the user of the library how they want to identify they thing that they're running. Because these things are a tree if you needed access to the feature, and not the scenario, you can get it via scenario.feature.

So basically it'd look like:

func shouldRun(scenario:Scenario, withTags: [String]) -> Bool {
    return scenario.position.line == 10
}

or if you're so inclined

func shouldRun(scenario:Scenario, withTags: [String]) -> Bool {
    return scenario.feature?.position.line == 10
}

Finally if you had gherkin something like this:

    Feature: Some terse yet descriptive text of what is desired
      Scenario Outline: the <title>
        Given the <data>

            Examples:
              | title  | data  |
              | first  | 1  |
              | second | 2  |

And you wanted to you could do

func shouldRun(scenario:Scenario, withTags: [String]) -> Bool {
    return scenario.title == "the first"
}

Thus giving a more solid hook into that particular scenario regardless of line number.

What do you think?

[Tyler-Keith-Thompson]

Also it seems like we might want a totally separate feature to "just rerun the things that failed" or perhaps a variant "run this number of times and only fail the test if it fails some % of the time".

[olyv]

if I understood you correctly, it would fine to have

func shouldRun(scenario:Scenario, withTags: [String]) -> Bool {
    return scenario.feature?.position.line == 10
    // where 10 is a line number of particular example or number of the example in the data table
}

but I am bit concerned about

func shouldRun(scenario:Scenario, withTags: [String]) -> Bool {
    return scenario.title == "the first"
}

because it will require user to know exact example needed for run. And I am not really sure if any cucumber port allows to use datatables to fill in scenario outline like Scenario Outline: the <title>. While this point is not relevant for Swift only projects, it can be troublesome in projects where you have different cucumber implementations but the same tests (features), for example an application implemented for android ios (which means you need different cucumbers for java and for swift)

[Tyler-Keith-Thompson]

So the whole token in a scenario outline thing should be part of the Cucumber standard and is already included in CucumberSwift as a feature.

Here's a link to the "good" testdata files that are used to test the AST: https://github.com/cucumber/cucumber/blob/master/gherkin/ruby/testdata/good/example_tokens_everywhere.feature

That particular one is for Ruby. Here's one for Java: https://github.com/cucumber/cucumber/blob/master/gherkin/java/testdata/good/example_tokens_everywhere.feature

So I think we should be good on the idea of sharing Gherkin files between different Gherkin implementations in different languages.

Regardless the idea of exposing the parsed Scenario object to the shouldRun method seems to be a reasonable approach for now.

[Tyler-Keith-Thompson]

Alright my friend I have a hacked together prototype that works with 1 test (yours)

Gherkin:

Feature: Some terse yet descriptive text of what is desired
    Scenario Outline: Some determinable business situation
        Given a <thing> with tags

        Examples:
            |  thing   |
            | scenario |
            | scenari0 | #line 8

Implementation:

public func shouldRunWith(scenario:Scenario?, tags: [String]) -> Bool {
    return scenario?.position.line == 8
}

If you're interested in trying it out you can load it into your podfile like so:

pod 'CucumberSwift', :git => 'git@github.com:Tyler-Keith-Thompson/CucumberSwift.git', :branch => 'line-numbers'

I'll keep plugging away at it this weekend and see if I can add more tests and refactor a bit.

If you do happen to give it a try let me know if it sort of feels natural to use, or if there's a better way you'd like to interact with the library to achieve your goals.

[olyv]

I don't have an access to my computer containing source code but I'll get back to this on Monday. Thanks!

[olyv]

Ok, I took me a bit longer to test it out but I got stuck with one weird thing. I tried to debug it but I guess I am still missing some parts of the parsing logic. The issue can be reproduced with the feature file like

@MyTestCase

Feature: Some terse yet descriptive text of what is desired

Scenario Outline: Some determinable business situation
 * a background step executed for each scenario with <thing> and <number>
 Given some stuff

Examples:
|  thing | number |
|  foo    |  1            |
|  bar    |  2           | #line 12

and if you try to execute it with

   public func shouldRunWith(scenario:Scenario?, tags: [String]) -> Bool {
      return scenario?.position.line == 12 && tags.contains("MyTestCase")
   }

public func setupSteps() {
      MatchAll("^a background step executed for each scenario with (.*) and (.*)$") { foo, _ in
         print(foo[1])
         print(foo[2])
      }
}

Then the feature is parsed into two scenarios with position.line properties 12 and 13. for example, for, there is a scenario parsed for first example where line number should be 11

Not really sure why it happens: I tried different combination of steps, changed * a background step to Given a background step assuming there was a problem with scope but still can't get what is wrong. Do you have any clue?

[olyv]

Now I am completely lost in advanceToNextToken() function but this is a guilty party. It's not parsing 'Given' step as expected -- it assigns different positions (line numbers to be precise) to 'Given' keyword and its match:

Please note: screenshot can be a bit confusing because it demonstrates the scenario I used for testing but the actual breakpoint is set at lex() function where I can see tokens parsed from feature.

[Tyler-Keith-Thompson]

Wow, excellent job tracking that down. I'm not actually particularly happy with the way I'm trying to capture lines/columns (columns are currently completely off).

I'll dig into what went wrong with this implementation and see if I can't clean it up a bit. The weird thing conceptually is that these are ultimately in ranges, so if you try to run on a line number that's in the middle of a scenario it should really probably just run the whole scenario.

Swift obviously supports ranges quite well I just need to think through a better way to handle them in this case and attach them to tokens with a more sane implementation

[Tyler-Keith-Thompson]

Alright I've got a much less "hacked" version now. The "good" test data from the official Cucumber repo actually has line numbers in their AST json. So I made sure to compare the output from the lexer against those and everything looks pretty decent at the moment.

I'm gonna futz with columns and ranges before I merge this into master, so it may take a little bit longer. Looks like we're about 17% less efficient with line numbers parsing 100 features with 2 scenarios each takes ~0.130 seconds. I can live with that.

Feel free to pull the branch and test again, your particular issue has been solved. I may end up changing the syntax for how you specify what line you want to run on, but for right now at least it's the same. I'll add a wiki page detailing how to run by line after I merge into master.

[Tyler-Keith-Thompson]

Update I did in fact change the syntax, it's now like this:

public func shouldRunWith(scenario:Scenario?, tags: [String]) -> Bool {
    return shouldRun(scenario?.withLine(4))
}

NOTE: return keyword is not necessary if you're running Swift 5 or greater

shouldRun is just a wrapper to turn an optional boolean into a boolean and is declared as a global function with CucumberSwift

The previous way of accessing this (with the .line property) is not recommended, as it does not check for ranges.

Still gonna futz with columns a little bit longer before I merge to master

[Tyler-Keith-Thompson]

Sorry for the spam. I've been in the zone today. I got columns figured out, verified by the official Cucumber AST output. I've got a few test cases in place, and I'm fairly happy with the readability/extensibility/performance of it all.

So this is available in CucumberSwift v2.2.9. I've updated the wiki here with details.

I'm gonna go ahead and close the issue for now. As always if you run into any problems open a new issue. I'll add some tests and get it fixed.

[olyv]

Great news! Thank you very much for tackling this.