Unit Test are too slow

[htdvisser]

Summary

While reviewing #2730 I noticed that some of our unit tests are really slow. Slow tests slow down development, so let's see if we can speed things up a bit.

Steps to Reproduce

Just check CI.

What do you see now?

Here's the top 10:

Package	Duration
pkg/gatewayserver	65.747
pkg/applicationserver	53.63
pkg/applicationserver/io/pubsub/provider/nats	20.546
pkg/gatewayserver/io/udp	15.849
pkg/networkserver	14.465
pkg/applicationserver/io/pubsub	10.511
pkg/applicationserver/io/pubsub/provider/mqtt	10.322
pkg/identityserver	8.578
pkg/identityserver/store	5.378
pkg/applicationserver/io/web	3.253

What do you want to see instead?

These tests don't all need to take this long.

Environment

CI

How do you propose to implement this?

• Let's start by looking at why some tests take so long
• Then we can see if it's really necessary for them to take so long
• Maybe we can run some test with t.Parallel()
• Maybe we can use the -short flag for some of the builds?

[rvolosatovs]

networkserver expects developers to use -short during development, go test runs the whole test suite and that will take time - there's no way around that, since we're dealing with e.g. RX delays in flow tests

[htdvisser]

I'm a bit disappointed that you don't want to take any responsibility here, @rvolosatovs. Can none of those test be run in parallel to speed things up? Can we really not use mocked time or something?

[rvolosatovs]

I'm a bit disappointed that you don't want to take any responsibility here, @rvolosatovs. Can none of those test be run in parallel to speed things up? Can we really not use mocked time or something?

There are several tests in NS that need to be rewritten, which will be done later - it's pointless to try to speed them up at this point of time. NS already uses mock time, hides slow tests behind a -short and runs tests in parallel. go test in Network Server will be slow by design - in flow tests we need to wait 5 seconds for the join-accept to be delivered alone. E.g. if NS has a flow test for OTAA device, which performs a join, then sends an uplink and receives a downlink, it will take at least 6 seconds. And we're talking about just one test case - full NS testing suite tests implementation against various PHY version, MAC version and frequency plan combinations. Please do not run go test during development, run go test -short -failfast instead - tests during development run fast.

I'm not sure where the "don't want to take any responsibility here" comes from and seems quite harsh. Please run go test -short locally and if that takes too much time in your opinion, please feel free to file an issue. 14 second-long NS tests in CI are reasonable and expected in context of Network Server.

[johanstokking]

pkg/applicationserver and pkg/gatewayserver have a similar test pattern; the frontends are tested as test cases sequentially. This is super useful, but this should be refactored to a separate test bench in pkg/{application|gateway}server/test that takes an interface. When fully isolated, they can run in parallel. For GS, this is already tracked in https://github.com/TheThingsNetwork/lorawan-stack/issues/1981.

[johanstokking]

Let's open more concrete issues on specific parts.