Testing Frameworks and Shell Script Elimination

[nelsam]

The Problem

Right now, our tests are run using environment variables to define which dialect is being tested, and which driver is being used for the connection to test against. This means that our CI runs tests with a shell script, and setting up a testing environment locally is inconvenient (especially in the cases where bash isn't available).

What I've Done Elsewhere

I started out using testify, long ago, and wrote a suite package for it. The big feature that the suite package added for my purposes was the ability to do the following:

func TestQueryPlanPostgres(t *testing.T) {
    dialect := gorp.PostgresDialect{}
    connection, err := sql.Open("postgres", postgresURL)
    if err != nil {
        t.Errorf("Could not connect to postgres: %s", err)
        return
    }
    suite.Run(t, PlanTestSuite{dialect: dialect, db: connection})
}

This further enabled me to test specific connections/dialects much easier:

$ go test -test.run '(MySql|Sqlite)'

Our Options (WIP)

This is rudimentary and incomplete - I'm just jotting thoughts down for now.

• ginkgo/gomega
  • +
  • Very clean BDD-style language
  • -nodot flag for avoiding dot-imports
  • -
  • Suite semantics are held internally, doesn't seem easy to run tests multiple times, against multiple dialects/connections
  • Dot imports by default
  • Doesn't seem like combining it with other testing systems would read very clean
• goconvey
  • +
  • Clean BDD-style language
  • Encourages you to use other testing systems in addition to goconvey
  • Very clean web UI for monitoring tests as you're working (and the web UI works with any tests supported by go test - not exclusive to goconvey tests)
  • -
  • Docs encourage dot imports
  • Doesn't have the suite-like logic we'd need in order to run the same tests against multiple dialects/connections
• gocheck
  • +
  • Has all the suite stuff that neither ginkgo/gomega nor goconvey support
  • -
  • Test language doesn't read like a spec
  • Might not be able to access the *testing.T value, make it difficult (impossible?) to use language from other testing frameworks/toolkits.

[GeertJohan]

I've tried some strategies. gocheck seemed nice at first, with the builtin support to run the suite with different contexts. But I found that there is no convenient way to write to logs in which context the test failed.

goconvey doesn't support contexts out of the box, but it's very easy to add them manually. However, then also the context is not displayed in the go test results. Could be done by adding a prefix on each context that is used..

So, I'm still not really satisfied with either solution.. What we could do too is using any of these above without requiring multiple contexts/dialects in the testing code. Then we'll add a tasks script (I like slurp) that runs the tests one time for each database we have. We'll need to be able to change the connection string for databases anyway, no-one has the same testing environment, so why bother with contexts IN the tests?

((Sidenote: I would like to have a slurp tasks script anyway to setup and test with databases in docker containers. That should make it really easy to setup, test and destroy with most databases without installing them on the local machine.))

I've created a tmp repo for these tryouts: https://github.com/GeertJohan/gorp-tests

[GeertJohan]

Work in progress on shell script alternative, the gorptool: https://github.com/GeertJohan/gorp/tree/gorptool/gorptool

At this time, it can run tests for mysql and postgres and can set up docker containers for mysql and postgres. (gorptool test all-docker, run in github.com/go-gorp/gorp folder)