Test Coverage and Integration Tests

[rfratto]

At the moment, Loki is tested solely through unit tests. These unit tests are the only way to validate changes for Loki, and the current coverage per package is generally low:

?       github.com/grafana/loki/cmd/docker-driver       [no test files]
ok      github.com/grafana/loki/cmd/logcli      0.026s  coverage: 13.1% of statements
?       github.com/grafana/loki/cmd/loki        [no test files]
?       github.com/grafana/loki/cmd/loki-canary [no test files]
?       github.com/grafana/loki/cmd/promtail    [no test files]
ok      github.com/grafana/loki/pkg/canary/comparator   0.217s  coverage: 84.0% of statements
?       github.com/grafana/loki/pkg/canary/reader       [no test files]
?       github.com/grafana/loki/pkg/canary/writer       [no test files]
ok      github.com/grafana/loki/pkg/chunkenc    4.654s  coverage: 66.8% of statements
ok      github.com/grafana/loki/pkg/distributor 0.036s  coverage: 60.0% of statements
?       github.com/grafana/loki/pkg/helpers     [no test files]
ok      github.com/grafana/loki/pkg/ingester    25.584s coverage: 65.7% of statements
?       github.com/grafana/loki/pkg/ingester/client     [no test files]
ok      github.com/grafana/loki/pkg/iter        0.054s  coverage: 55.0% of statements
ok      github.com/grafana/loki/pkg/logcli/output       0.017s  coverage: 91.7% of statements
?       github.com/grafana/loki/pkg/logentry/metric     [no test files]
ok      github.com/grafana/loki/pkg/logentry/stages     0.073s  coverage: 66.6% of statements
?       github.com/grafana/loki/pkg/logproto    [no test files]
ok      github.com/grafana/loki/pkg/logql       0.035s  coverage: 87.6% of statements
ok      github.com/grafana/loki/pkg/loki        0.052s  coverage: 12.8% of statements
ok      github.com/grafana/loki/pkg/promtail    6.681s  coverage: 80.0% of statements
ok      github.com/grafana/loki/pkg/promtail/api        0.004s  coverage: 39.0% of statements
ok      github.com/grafana/loki/pkg/promtail/client     0.402s  coverage: 79.3% of statements
?       github.com/grafana/loki/pkg/promtail/client/fake        [no test files]
?       github.com/grafana/loki/pkg/promtail/config     [no test files]
?       github.com/grafana/loki/pkg/promtail/positions  [no test files]
ok      github.com/grafana/loki/pkg/promtail/scrape     0.027s  coverage: 62.5% of statements
?       github.com/grafana/loki/pkg/promtail/server     [no test files]
?       github.com/grafana/loki/pkg/promtail/server/ui  [no test files]
ok      github.com/grafana/loki/pkg/promtail/targets    2.055s  coverage: 42.1% of statements
ok      github.com/grafana/loki/pkg/querier     0.134s  coverage: 36.5% of statements
ok      github.com/grafana/loki/pkg/storage     0.034s  coverage: 90.2% of statements
?       github.com/grafana/loki/pkg/storage/hack        [no test files]
ok      github.com/grafana/loki/pkg/util        0.016s  coverage: 1.6% of statements
?       github.com/grafana/loki/pkg/util/flagext        [no test files]

Loki needs higher test coverage on both the unit and integration testing level to help add confidence to developers making changes.

[pracucci]

I agree. I would also suggest to introduce a tests requirement policy on PRs. I'm seeing most of the changes introduced in PRs without any test at all.

[rfratto]

Requiring new features to include test coverage is a good idea.

(Even though I think I'm also guilty of introducing new functionality without tests :slightly_smiling_face:)

[slim-bean]

I don't think this needs to be kept open as an issue, we always want to have good test coverage and this is an ongoing conversation.