Extend TestRunFinished with basic stats

[pranas]

First of all, great project! I was able to build a basic Cucumber runner/frontend on top of it in a day (check out pranas/cucumber-go).

I'm working on formatters to display some nicer output and noticed TestRunFinished response is rather basic. Pretty much all implementations and all formatters will show summaries like:

588 steps (570 passed, 4 failed, 3 pending, 1 undefined, 10 skipped)

It would be nice to get that back from cucumber-engine.

[aslakhellesoy]

Hey @pranas - that's some impressive work you've done there!! Kudos to you and to @charlierudolph for building an engine that others apparently can interface with easily!

A few questions:

• Why did you copy the dots-formatter code? We'd like that to be a reusable library (as well as a command-line tool). How can we improve it so you could use it as a lib?
• I know it's convenient to interface with cucumber-engine via Go function calls. However, I really would like to see this interfacing happen over TCP/protobuf - for a couple of reasons:
  • a) It would make it possible to run distributed
  • b) It would increase our confidence that the protocol is sound

I agree with you that the Summary should come from the engine (as a protobuf message).

@pranas - would you consider donating this to the cucumber project? I know a few people who would jump on this to help out!

[pranas]

@aslakhellesoy I would be happy to donate. I also have some more ideas/work I want to put into it. My goal was to enable scenario concurrency on my Cucumber test suite (no other runner for Go could do it). Now I want to make it a little more friendly for day-to-day usage (formatters, filtering and etc).

Why did you copy the dots-formatter code? We'd like that to be a reusable library (as well as a command-line tool). How can we improve it so you could use it as a lib?

I was going to use the dots-formatter as dependency. I even had an initial interface that worked with original code untouched via IO pipes, but I still had to copy the code because there was some issues with the go.mod in that project (maybe package name mismatch in monorepo? can't remember exactly). Also, basic dots are pretty useless without summary and failed steps list so it deviated even further. By the way, in order to display failed steps in a formatter you have to have a lot of tracking going on, maybe this could also be a part of TestRunFinished message.

Eventually, I would like to contribute back and switch to using the dots library but first I want to have this usable and figure out how the final layout should look like.

I know it's convenient to interface with cucumber-engine via Go function calls. However, I really would like to see this interfacing happen over TCP/protobuf - for a couple of reasons:

It shouldn't be too hard to replace channels with IO.

[pranas]

I've created summary formatter which prints output like this:

Failed steps:

  Scenario: 2+2 # features/calculator.feature:94
    displays "4" # features/calculator.feature:96
      Error: expected 4 got 5

20 scenarios (18 passed, 2 failed)
138 steps (128 passed, 2 failed, 8 skipped)
0.768558037s

You can find it here. I think a lot of this tracking should go in cucumber-engine to make formatters dumber. Besides the scenario/step counting, it's rather painful to display failed steps (you have to keep track of all pickles so you can return all the necessary data once you detect failed step).