Evaluate BDD testing frameworks for our API

[Martheos]

Ginkgo seems like a viable option

[BenedictHomuth]

BDD Test Framework Evaluation

Behaviour-Driven Development (BDD) can be seen as a way to bridge the gap between the software and business people in regards to communicating desired behaviors of a system. It is encouraging collaboration across roles and fosters a shared understanding of the problem to solve.

Ideally, it creates a human-readable file – often expressed in gherkin syntax.

• Absolute basic should be Given, When, Then format.

Example gherkin syntax:

Feature: User Login

  Scenario: Successful login with valid credentials
    Given I am on the login page
    When I enter "username" as the username
    And I enter "password" as the password
    Then I should be logged in to the system

Contenders

Build for BDD

• gobdd (⭐️ 107)
• godog (⭐️ 2.2k)

Testframeworks which support BDD

• ginkgo (⭐️ 7.9k)
• goconvey (⭐️ 8.1k)

Other / Related

• goblin (⭐️ 886)
• gogiven (⭐️ 14)
• gospecify (⭐️ 53)
• biff (⭐️ 14)
• gherkingen (⭐️ 68)
• testcase (⭐️ 116)
• apitest (⭐️ 738)

Tryout

godog

• Hardest to get started from the three options.
• Feels a lot like magic, since it compiles the .feature files you write using "gherkin"-style into function boilerplates
• Hard to figure out how to phrase the gherkin text in such a way, that the boilerplate generator spits out the form that you want. (Maybe my own inexperience)
• Could get messy, when there is lots of stuff to test (I found it hard to structure)
• BUT: The test must always be up to date with the written feature file, otherwise the test breaks, which is good for staying in sync between requirements and tests

ginkgo

• Very easy to get started.
• Not adhering to the gherkin syntax
• Therefore it is a general-purpose testing framework, that expresses tests close to natural language
• Has a companion package gomega for simplified test result matching (That thing is changing lives! Also feels like the postman tests)

goconvey

• Easy to get started
• Matching is not as smooth as with gomega
• Has a webUI, that reruns all tests on file change.
• The webUI has a gherkin to test-boilerplate composer which works great – or at least is easier to use than that from godog
• The requirements are not connected to the tests. This way there is no sync between changes in requirements or tests

Conlusion

There can always be a debate on which framework is better for a specific purpose. Nonetheless, I find it important to go for frameworks that are already very stable and have a community behind them, that will maintain it going forward. With this being the minimum criteria, from all the listed tools only three remained: godog, ginkgo, and goconvey!

Out of those, godog could be dismissed, since it felt too clunky to handle. The only positive thing to say is, that it forces you to have your requirements listed in .feature files, and as soon as you change up something either in a test or feature file, the test breaks.

The remaining two are depending on our team's preference:

If we really want to go for the BDD spirit, I would suggest going with goconvey, since it comes closest to representing the gherkin-syntax in test cases. Additionally, it comes with a WebUI where you can write a gherkin text, and the test code is generated for you. This would give non-technical people the possibility to describe their requirements themselves.

On the other hand the by far superior testing framework (including matching results) here was ginkgo. It is easy to pick up, tests are readable almost like natural language and the gomega matching library most often used with ginkgo is also very intuitive. The downside is that there is not a built-in gherkin to test-boilerplate editor or a way to sync specified requirements with tests on changes in either of those files.