Reverse test generation from feature files

[mkschulze]

Hey,

if I understood correctly and in case I remember right from our last meeting, there could be a way to generate our Julia test cases from the feature files automatically with this package?

That would be a really great feature I think, it would allow us to move faster and keep ahead with Grakn development in the future more easy. Would that be much work to get going, if possible?

thx ;)

[erikedin]

There's a work in progress already available on master. You can run it like

$ julia --project=. /path/to/ExecutableSpecifications.jl/cmd/suggeststeps.jl /path/to/feature/file.feature features/steps

It will print suggested step implementations. The first argument /path/to/feature/file.feature is the feature file you want suggested steps for. The second argument features/steps is the path to the existing step implementations. The script will find all scenario steps that do not have a matching step implementation, and print it. For instance, if I run it on the connection/session.feature locally, I get

$ julia --project=. ../ExecutableSpecifications.jl/cmd/suggeststeps.jl features/connection/session.feature features/steps
using ExecutableSpecifications

@when("connection create database: grakn") do context
    @fail "Implement me"
end

@when("connection open session for database: grakn") do context
    @fail "Implement me"
end

@then("session is null: false") do context
    @fail "Implement me"
end

@then("session is open: true") do context
    @fail "Implement me"
end

@then("session has database: grakn") do context
    @fail "Implement me"
end
[ ... more here ... ]

Nothing of this is documented yet, unfortunately.

Also, I need to warn you that these are not perfect. I've generated these locally and looking for issues, and I've already found a couple, and fixed some. For instance, it seems like some of the Scenario Outlines aren't run properly, so I'm looking into that.

Another issue is that you will not get any organization of step implementations for free. I suggest you use this as a way to save typing, rather than using them as the one correct way to write the step implementations. For instance, the script will not be able to suggest any parametrized step implementations, but you will instead need to identify those yourself. My suggestion is to generate these implementations slowly, no more than a feature file at a time. Otherwise you'll just end up with a thousand step implementations that are impossible to organize.

[mkschulze]

This is a really exciting feature @erikedin I think. I will give it a go this way.

[mkschulze]

How do I set the ParseOptions to have more lenient parsing @erikedin?

I just tried to parse a file and got:

using ExecutableSpecifications
parseOpt = ParseOptions(allow_any_step_order=true)
suggestmissingsteps("./test/behaviour/features/concept/thing/attribute.feature", "./test/behaviour/features/steps", parseOpt)

UndefVarError: ParseOptions not defined

Stacktrace:
 [1] top-level scope
   @ In[45]:2
 [2] eval
   @ ./boot.jl:360 [inlined]
 [3] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
   @ Base ./loading.jl:1094

[mkschulze]

can be closed meanwhile I think.