Flow tests in components

[rvolosatovs]

Summary:

Apart from isolated unit tests should contain flow tests, where various common/expected flows are tested. For example in NS context that could be an example:

1. Create a device via Set(possibly multiple devices)
2. Call HandleJoin
3. Call HandleUplink
4. Call ApplicationDownlinkPush
5. Call HandleUplink

Some rules:

• The test package name must be <package-under-test>_test and the tests placed at the same directory as the package under test. For example, if package A is tested, the name of the package containing flow tests must be A_test.

• The name of test file containing flow tests for package A must be A_flow_test.go.

• All RPCs should be called on the component via the generated RPC client and no non-exported entities from the package under test should be used.

• Components should use mocks for external components if their functionality is necessary. For example, in NS example above - if GS, AS, JS or some other component functionality is required - the testcase would use a mock GS/AS/JS etc.

• Components should use actual implementation(s) for internal components(anything defined in the package under test or at levels below, e.g. if package A is tested, functionality in A.B and A.C should not be mocked). If multiple implementations of some interface are possible, the flow test should test all implementations by ranging over them in a loop. (See https://github.com/TheThingsNetwork/lorawan-stack/blob/f1377b316a85ae3709a84db5b5f9a46faafba3a0/pkg/networkserver/registry_test.go#L245-L281 for an example of how to do this) For example, in NS example above - Redis implementation of device registry defined at pkg/networkserver/redis should be used. Note, that if there was a different implementation of the device registry present in NS, that would have to be tested via the same test function.

Note, that this approach to testing does not replace isolated white-box unit testing and both are equally important.

Why do we need this?

To ensure our components do what we(and our users) expected them to

What is already there? What do you see now?

https://github.com/TheThingsNetwork/lorawan-stack/blob/f1377b316a85ae3709a84db5b5f9a46faafba3a0/pkg/networkserver/registry_test.go#L34-L235 is an example of a "flow test", but on a lower level and it only tests one flow. We need something similar, but on component level and testing several distinct flows.

What is missing? What do you want to see?

Flow tests

How do you propose to implement this?

Write/adapt the tests described

[rvolosatovs]

@johanstokking @htdvisser please look through the proposal and give your opinion

[htdvisser]

Apart from the fact that your summary contains the content I'd expect in "What is missing" and "How do you propose to implement this", this sounds fine to me. Of course we would have to skip tests of infeasible implementations (just like we don't test S3 buckets in the blob package if there is no configuration for it).

[johanstokking]

I find this important enough to scope this to NS and assign the core team. This requires also in depth understanding of LoRaWAN, NS, the way NS is tested now, etc, to call for community help.

Keeping it in Backlog now, but we need to have it, if only for automated regression testing.

Suggesting to put AS in here as well, but let's keep this flow testing and not integration testing.