Project collab

[mkschulze]

Hello,

a few people aim to set up a community project in the BDD way in Julia.

It may be nice to use this package, but it seems there is no further development going on? Would you mind sharing your ideas and if you are going to get back to this?

thank! Mark

[erikedin]

Hi!

I plan to pick this up again. Just yesterday I verified that the tests still pass for Julia 1.5.3. Unfortunately, I can't guarantee that I will have a lot of time to work on it. It's a hobby project only for me, so I'll only work on it in my free time.

I think that the major missing feature is variables. You can work around most other things, but missing variables in step definitions, like

@given "some value {foo}" begin
   ...
end

is a problem. Without variables you'll have to copy/paste a lot of step definitions like these.

Another problem for you, if you're planning on starting to use it, is that there's no support yet for selecting which scenarios and features to run based on tags. I mean that you'd probably want to mark scenarios with @wip and tags like it. There's support for reading tags from the Gherkin files, but they're not used yet (as far as I can remember). That's going to make it more cumbersome to use for you.

I guess, basically, that I wouldn't recommend that you depend on this package for a new project. It might be good enough in the near future, but probably not for at least a few months, and even that is far from guaranteed.

[mkschulze]

I see, ok. Thanks for the quick response and explanation!

[mkschulze]

Actually, we might be interested to support you and contribute to the repo, in order to get the specification and the above mentioned key features in. But we would need some guidance where to start and what to do. Is this aimed to be compatible with cucumber eventually?

[erikedin]

Regarding Cucumber, the syntax of the Gherkin files will be the same (or very, very close), but the implementation of those scenarios will not be compatible with Cucumber. The reason is mainly that I want this to have very few dependencies, and something people should be able to run as part of the test/runtests.jl suite. That said, it may be possible to maintain Cucumber compatibility. I've honestly never used Cucumber extensively, so I'm not sure what would be required.

As for where to start, that's something I'll have to get back to you with. I've barely looked at this code for two years, so I'm in a similar position myself. I'll try to write an overview of the code, from a design perspective, and maybe that will be of help.

[mkschulze]

Ok, so let me find a few words on what we try to achieve and what we are doing. I'm part of the Humans of Julia org here on GitHub and we want to port over a client interface to a Graph database called Grakn. The package we prepare to work on is called GraknClient.jl

So, they at Grakn are following the BDD approach all over their suite using this behaviour repo. It would be great to make use of that to guarantee the same behaviour of our port.

We want to eventually clone this behaviour from GitHub, get the feature file we want to test, get the step implementation files we want to use, and run a test runner. However, we will need to find a Gherkin runner for Julia - or at least one that can make use of GraknClient.jl.

I could also very well see such a Gherkin implementation for another Julia project as well actually.

However, I saw an old issue in the cucumber repo where this came up already and just recently opened a new one here. I'm honestly not sure how difficult it would be to start over and implement the Gherkin specifications or if we could better put effort in your repo and maintain it together maybe. I think I would prefer the latter to be honest.

Maybe I could start translating this repo into Julia lang: https://github.com/cucumber/gherkin-python and then we could implement that in your runner for example?

[mkschulze]

Cucumber compatiility itself wouldn't be required actually I think. Would just be great to have a Gherkin parser and runner, like behave.

[erikedin]

Regarding the https://github.com/cucumber/gherkin-python repo, as I understand it, it's a Cucumber-specific thing. It wouldn't make sense in this repo, since I did my own Gherkin parser. That repo seems to read the internal representation Cucumber creates from feature files. I may have misunderstood it, but that's the impression I got.

A thing I noticed in the Grakn behaviour repo is that they often write Given/When/Then steps out-of-order. For instance,

Scenario: one database, one session, one committed write transaction is closed
    When connection create database: grakn
    Given connection open session for database: grakn
    When session opens transaction of type: write
    Then session transaction commits
    Then session transaction commits; throws exception

So above we have a When step before a Given step. I disallowed that in my parser. We'll have to ensure that my parser can read those files properly. Here's a test I have that verifies that the situation above is disallowed: https://github.com/erikedin/ExecutableSpecifications.jl/blob/e2ead43fb3b49ac21c7febcb328efe2fb1cc62e0/test/gherkin/scenario_test.jl#L128 For this situation specifically, it's pretty easy to fix I'd say. We'll add an option to allow more lenient parsing. I don't think it's a major thing, but it's something we'll have to fix.

Another Gherkin syntax thing that I saw was tables like

Then some step
  | cell 1 | cell 2 |
  | cell 3 | cell 3 |

I'm not sure that I support that yet, but we'll add it if it's missing.

[mkschulze]

Interesting ok, makes sense.

Currently I'm trying to set up a BDD plugin for the Genie webframework in Julia, based on your repo and Coffee machine example.

This could end up in a stipple.app then maybe.

Would be a nice visual tool, like cucumbers react thing, just in Julia/Vue.js

[mkschulze]

@erikedin could you elaborate a little on how the variable support would ideally be integrated, like where to start and what to take into consideration?

[erikedin]

So, I haven't given it much specific thought, but here's some ideas. First of all, I think a reasonable syntax for the variables is

@given "some value {foo}" begin
   @expect foo == 17
end

but that's not a hard requirement. If some other syntax is easier to implement, then go for it. I'm not sure if there are any Julia standard library functions that does matching like this, or if we could somehow re-use string-interpolation for it. But writing variables like {foo} seems reasonable to me. So, if you were to write a Gherkin step like

Given some value 17

then that would match the above code. Now, maybe we can make the foo variable directly accessible, like I wrote above. However, I'm not even sure we can do that. I haven't looked at Julia macros in forever, so I'm not sure. Perhaps we'll need to make them accessible some other way.

As for implementation, I'm also a bit fuzzy on the details. When we execute a scenario, then for each step (Given/When/Then), we find a definition that matches. That code is at https://github.com/erikedin/ExecutableSpecifications.jl/blob/master/src/executor.jl#L84-L93 That just calls the findstepdefinitions method, which is the one that should see that the Gherkin step Given some value 17 would match @given "some value {foo}" begin .... end.

Today, the findstepdefinitions method just does an equals. https://github.com/erikedin/ExecutableSpecifications.jl/blob/master/src/stepdefinitions.jl#L133-L140 If the text in the Gherkin step equals that of the definition, then there's a match. For definitions with variables, we would have to do some more matching there, and ensure that all variables are bound to a value. That method returns a StepDefinition type (defined in the same file). I guess we either make another StepDefinitionWithVariables type, or we add variable support to the existing StepDefinition type. Not sure about that.

We invoke the actual step definition here https://github.com/erikedin/ExecutableSpecifications.jl/blob/master/src/executor.jl#L101 The actual macro that runs the defined step is here https://github.com/erikedin/ExecutableSpecifications.jl/blob/master/src/stepdefinitions.jl#L64-L74 Somehow that should make the all the variables available to the code in the @given .. begin .. end block. I haven't thought about this much.

I'm not sure how understandable all of the above is.

[mkschulze]

Well, yes that is good information, thx! Technically, I think @tk3369 would be able to share some thoughts on the options Julia would provide.

[tk3369]

Thanks @erikedin and @mkschulze, great discussions! I've been busy lately and I'm just catching up today. Let me take a deeper look and come back to this.

[tk3369]

I have looked into a bunch of feature files from Grakn's behaviour repo and played with ExecutableSpecifications.jl a little bit. I think a sensible approach would be to first document the gaps and then we can plan and tackle each one.

Until we have a better place to store this document, I have pasted it below. We can use it for creating new tracking issues.

Assessment for Grakn client implementation

Summary of Gherkins features

The following Gherkins keywords are found at the official Cucumber site. Features supported by ExecutableSpecifications.jl are checked accordingly.

Primary keywords

• [x] Feature
• [ ] Rule (as of Gherkin 6)
• [x] Example (or Scenario)
• [x] Given, When, Then
• [x] And, But for steps (or *)
• [ ] Background
• [x] Scenario Outline (or Scenario Template)
• [x] Examples (or Scenarios)

Secondary keywords

• [ ] """ (Doc Strings)
• [ ] | (Data Tables)*
• [x] @ (Tags)
• [ ] # (Comments)

Notes: (*) Data tables is partially supported in the Examples section of scenario outline.

Features required for implementing Grakn client

Background keyword

The Background feature is used extensively. It is used to run multiple Given statements before executing each scenario.

Doc strings

Not to be confused with Julia doc strings. It is a multi-line string that is passed to the step's context in the text attribute. See behave's documentation.

Example:

  Scenario: write data in a schema session throws
    When connection create database: grakn
    Given connection open schema session for database: grakn
    When session opens transaction of type: write
    Then graql define
      """
      define person sub entity;
      """
    Then graql insert; throws exception containing "session type does not allow"
      """
      insert $x isa person;
      """

Data tables

Data tables are used extensively. It appears that implementation differs in language bindings. For example, the Java implementation maps to a number of data structures. In Python, the behave package simply returns an object that implements some kind of Table API.

Example:

  Scenario: for many databases, open many sessions
    When connection create databases:
      | alice   |
      | bob     |
      | charlie |
      | dylan   |
      | eve     |
      | frank   |

    When connection open sessions for databases:
      | alice   |
      | bob     |
      | charlie |
      | dylan   |
      | eve     |
      | frank   |

    Then sessions are null: false
    Then sessions are open: true
    Then sessions have databases:
      | alice   |
      | bob     |
      | charlie |
      | dylan   |
      | eve     |
      | frank   |

Comments

Comments exists throughout the Grakn feature files for documentation purpose.

#    Given all answers are correct in reasoned database
    # There are 3^4 possible choices for the set {$x, $y, $z1, $z2}, for a total of 81
    Given answer size in reasoned database is: 81

Out of order steps

Sometimes When is provided before Given. It seems to be backward and not a good practice to provide the trigger prior to listing the pre-conditions? Perhaps we should talk to the Grakn team to fix it on their side.

Example:

  Scenario: an abstract entity type can be defined
    When graql define
      """
      define animal sub entity, abstract;
      """
    Then transaction commits
    Then the integrity is validated
    Given session opens transaction of type: read
    When get answers of graql match
      """
      match $x type animal; $x abstract;
      """
    Then uniquely identify answer concepts
      | x            |
      | label:animal |

Interleaving When and Then's

Grakn's features definitions are long and often embedding multiple, interleaving When's and Then's. It seems to be a bad practice. However, asking them to fix is probably going to be a huge undertaking. We should probably support that regardless.

  Scenario: Relation with role players can be created and role players can be retrieved
    When $m = relation(marriage) create new instance with key(license): m
    Then relation $m is null: false
    Then relation $m has type: marriage
    Then relation(marriage) get instances contain: $m
    When $a = entity(person) create new instance with key(username): alice
    When $b = entity(person) create new instance with key(username): bob
    When relation $m add player for role(wife): $a
    When relation $m add player for role(husband): $b
    Then relation $m is null: false
    Then relation $m has type: marriage

Blank lines

Sometimes there are additional blank lines between When's and Then's in the same scenario. The python behave parser seems to be more relaxed than ExecutableSpecifications.jl at the moment.

Reference: https://github.com/Humans-of-Julia/GraknClient.jl/issues/22

[mkschulze]

Great post @tk3369!

I actually had some talks a few days ago with Grakn about the ordering of When/Given/Then. I offered to look at the files and come up with proper ordering, but they went against this convention for now. Also because they are in full release mode currently I guess.

if you want to switch it around please do discuss in advance some of our scenarios are 20+ steps so sticking to one order is not going to necessarily look nice! look on a case-by-case basis and see if it would make better sense if written differently because as sayd, some of our scenarios have many steps and many assertions, which may not necessarily be very cucumbery but it is how we want to keep them

But they also proposed a required style guide for some time later this year, because they also see mistakes lying around.

I gave the example from here:

Scenario: one database, one session, one committed write transaction is closed
    When connection create database: grakn
    Given connection open session for database: grakn
    When session opens transaction of type: write
    Then session transaction commits
    Then session transaction commits; throws exception

Which would better be:

Scenario: one database, one session, one committed write transaction is closed
    Given connection create database: grakn
    Given connection open session for database: grakn
    When session opens transaction of type: write
    Then session transaction commits
    Then session transaction commits; throws exception

So yes, we might want to relax this test case for the time being.

[erikedin]

Thank you for this, it's very clear what's missing. Regarding doc strings above, I think the Gherkin parser supports it, but I'm not sure if it's missing somewhere else. I have tests on dock strings anyway: https://github.com/erikedin/ExecutableSpecifications.jl/blob/a0a17c8ec4f692ee41b14046d40116ad24082aa2/test/gherkin/scenario_test.jl#L207-L227 I'm not sure it's provided in the context though. I'll look into that. Perhaps it's only partially implemented.

My feeling when writing the Gherkin parser was that it wasn't very difficult. I have decent tests on it, and it's all line based and fairly simple, so the parsing itself shouldn't be hard.

[erikedin]

I think doc strings are implemented, but they're called "block text" in my code. This test exercises that bit of code. I'm not sure how clear it is, but essentially you should be able to read the doc string from the context, using the :block_text symbol, like so

@given "Something" begin
    s = context[:block_text]
end

    @testset "Block text" begin
        @testset "Scenario step has a block text; Context contains the block text" begin
            given = Given("Some precondition", block_text="Some block text")
            function check_block_text_step_definition(context::StepDefinitionContext)
                if context[:block_text] == "Some block text"
                    ExecutableSpecifications.SuccessfulStepExecution()
                else
                    ExecutableSpecifications.StepFailed("")
                end
            end
            stepdefmatcher = FakeStepDefinitionMatcher(Dict(given => check_block_text_step_definition))
            executor = ExecutableSpecifications.Executor(stepdefmatcher)
            scenario = Scenario("Description", [], [given])

            scenarioresult = ExecutableSpecifications.executescenario(executor, scenario)

            @test isa(scenarioresult.steps[1], ExecutableSpecifications.SuccessfulStepExecution)
        end

I think "block text" is my own term for it, so I think I probably ought to rename it "doc strings", to make it consistent with the Gherkin reference.

[erikedin]

I decided to get started on the lenient parser, because I think it's actually a fairly simple thing. I created a feature branch feature/lenientparser (it should be short lived), and made a very WIP commit, that allows for Given steps in any place. The commit is here https://github.com/erikedin/ExecutableSpecifications.jl/commit/57f019d64006686de3d0502b3572994c3e5799cf if you want to check it out. Basically, the parser is given a struct with a set of options for how to modify the parsing. Now it has a single option allow_any_step_order.

I have a question about using GitHub. I haven't used GitHub in a long time, and I'm unsure what features are available, except for git. I created some "Projects" earlier, to track different things I wanted to get done. Should I create one of those for this collaboration, to track progress?

[mkschulze]

A Project would make sense here yes, we use that for our repo as well. We named it according to the next goal, like version Nr. 1, and then link the commits and PR to it. Also in combination with the automatic Project Kanban here then, it leads to a nice overview.

[erikedin]

So I created this project https://github.com/erikedin/ExecutableSpecifications.jl/projects/5 and I started taking care of some easy tasks. The keywords But and * ought to work now, which was really simple, as I think they're just aliases for And. I've also implemented an option for a more lenient parser, so the Given/When/Then order shouldn't be a problem anymore. I'll keep trying to pick some low hanging fruit, just to get some progress.

[mkschulze]

very nice @erikedin! I wonder if we could find a better place for Toms list as a roadmap, or if we may create a PR or single issues out of it.

[erikedin]

I'm very open to suggestions. What kind of roadmap would you suggest? Like a Markdown file we commit to the repo, or does GitHub have any such roadmap features available?

[mkschulze]

Maybe we could create a graknclient-dev branch and put Toms list into a markdownfile or even the read.me. Then create a draft PR on that branch and merge other features branches into that dev branch, before merging into master eventually.

[mkschulze]

We may also create a CI at some point letting your tests run automatically.

[erikedin]

Sure, sounds good. I don't think any unrelated work will happen on master, so you can work against master as well, if you want, but perhaps a dev branch works better for you. Would you like me to create that branch for you, or will you make a PR?

[mkschulze]

Ok, maybe let's wait what @tk3369 would suggest as well before taking action. He has more experience than me. The package is not yet registered, is it? At least I couldn't install it via pkg> add ExecutableSpecifications

It could be wise to rebuild it based on the pkgtemplate.jl to also have ci, docs etc and I could make a logo as well.

[erikedin]

No, it's not registered anymore. It used to be, but long ago before the package system was overhauled in Julia. I'll have a look at PkgTemplate.

[tk3369]

Hi guys, sorry for being late. I've been very busy lately.

@erikedin Does :block_text work with When and Then keywords? They're used extensively in Grakn feature files.

Yeah, it would be nice to track these outstanding items. It seems a good idea to just put them up in the project https://github.com/erikedin/ExecutableSpecifications.jl/projects/5

As far as roadmap goes, I've seen people putting up a "mega" issue with checkboxes. When things get done they're just checked. Want to give that a try?

[erikedin]

Yeah, a mega tracking issue would work fine for me, I think. Would one of you like to create it? I imagine that if I create it, then you will not have permissions to check the boxes. I just tried, and it seems like I do have permissions to check boxes in task lists created by you (because I own the repo I assume).

@tk3369 Doc strings (:block_text) should absolutely work with When/Then too. I did not have an example of doc strings apparently, so I added one in example/features/spec2.feature and example/features/steps/steps2.jl. I realized just now that the output (when running the features) of those steps do not show the doc strings, so that's another thing I'll add to the issue.

[mkschulze]

Hi @erikedin,

we would like to invite you to our Discord channel if you like. We could have a chat and discuss things more dynamically maybe and get to know each other. We also have a video session tomorrow at 8 pm CET. We do this on a regular basis now, kinda like every three weeks to talk about plans, achievments and to make this a living project. In case you are interested, we would be happy to see you there as well at one of those dates. Here is the link: https://discord.gg/C5h9D4j

We are in the #graknclient.jl channel.

[erikedin]

Sorry, I missed the last message and saw it just today. I'd be happy to join it, but it might not be until later this weekend.

[mkschulze]

All good, our next video session is on the 28th of Febuary, but if you like to have a chat before that would be nice. Good to see you then!

[hakanai]

Regarding variables, the proposed syntax above was:

@given "some value {foo}" begin
   @expect foo == 17
end

In Cucumber, the bit inside the {} is actually a type and not a name, and you can define custom types by declaring a pattern and a function to parse it.

So I had imagined that future support for types might actually look like:

@given "some value {Int32}" do context, foo
   @expect foo == 17
end

But it's still not quite right, because nothing declares what characters to absorb into an Int32. I guess for built-in types this isn't much of an issue, but here's one I have in a Kotlin project using Cucumber:

        ParameterType("tuple", "tuple\\(($realRegex),\\s*($realRegex),\\s*($realRegex),\\s*($realRegex)\\)") {
                s1: String, s2: String, s3: String, s4: String ->
            Tuple(
                realFromString(s1),
                realFromString(s2),
                realFromString(s3),
                realFromString(s4)
            )
        }

So in my step definitions I can now put in {tuple}, while in steps, I can put in tuple(1, 2, 3, 4), and the values get passed to this lambda to assemble the final value. In this case, {real} is also a custom type, as the book (The Ray Tracer Challenge) uses all manner of different ways to declare real values, and I wanted to keep my specs verbatim to what was in the book where possible.

Types in Cucumber are typically spelled like a variable with a lowercase letter at the front, despite Cucumber itself arising from languages which would typically uppercase type names.