Integrate sub-tests into integration test framework

[AndrewSisley]

A good number of our tests use large blocks of repetitive data that causes copy-paste readability issues as well as wasting computing resources when performing multiple times.

We can hide the Go test syntax and other ugliness behind out test harness (https://go.dev/blog/subtests).

It is important that it is easy to run specific sub-tests, we wish to avoid the annoying situation where sub-tests must be commented out in order to run a single sub-test (use of the SubTest.Name property in the example, or similar, should hopefully allow this).

This may increase the likelihood of us needlessly coupling tests together, and the likelihood of writing tests that depend on state caused by a previous (we can perhaps randomise the order in which sub-tests execute to reduce this last problem).

For example, a test may look something like:

    test := testUtils.TestCase{
        Description: "Combination of a filter on regular and of an indexed field",
        Actions: []any{
            testUtils.SchemaUpdate{
                Schema: `
                        type User {
                            name: String @index
                            verified: Boolean
                        }`,
            },
            testUtils.CreateDoc{
                Doc: `{
                        "name": "John",
                                                "verified": true
                    }`,
            },
        },
        SubTests: []testUtils.SubTest{
            {
                Name: "Test name",
                Actions: []any{
                    testUtils.Request{
                        Request: `query {
                        Users {
                            name
                        }
                    }`,
                        Results: []map[string]any{
                            {
                                "name": "John",
                            },
                        },
                    },
                },
            },
            {
                Name: "Test verified",
                Actions: []any{
                    testUtils.Request{
                        Request: `query {
                        Users {
                            verified
                        }
                    }`,
                        Results: []map[string]any{
                            {
                                "verified": true,
                            },
                        },
                    },
                },
            },
        },
    }

The SchemaUpdate and CreateDoc would be run once, followed by the two sub tests (both Requests).

Idea voiced by Fred in our one-one, I'm just writing it up because I want a ticket for it.

[islamaliev]

I think it would be a nice feature.

Just throwing couple more ideas:

• we should execute them in random order by default, but would be nice to be able to set a flag Ordered to make sub tests run in the given order.
• we can add to each subtest flags Skip and Focus (or as an enum) to ease troubleshooting.

[AndrewSisley]

able to set a flag Ordered to make sub tests run in the given order

Why would you want this? Why should we permit this?

we can add to each subtest flags Skip and Focus (or as an enum) to ease troubleshooting.

Sub tests can be specified without such flags, using regex just like top level tests, see https://go.dev/blog/subtests for more info - I would not have been sold on this idea without this :)

[islamaliev]

Why would you want this? Why should we permit this?

Most of the time you don't need it, but I can tell from my personal experience that sometimes it's quite handy. We certainly don't need to bother with it right now. It's an option that we can think about once we have a real use case.

Sub tests can be specified without such flags, using regex just like top level tests, see https://go.dev/blog/subtests for more info - I would not have been sold on this idea without this :)

Yes, if you run them with terminal. Some people (including me) use more often IDE Not sure though if changing the code just to facilitate some IDE feature is a good idea.

[AndrewSisley]

Some people (including me) use more often IDE

Ah true, we definitely shouldn't make that painful, might be well worth adding a couple of easily settable props like you suggested then :)