Add performance regression test

[ernado]

We should check that MR will not significantly regress linter performance, like:

• CPU usage
• Memory usage
• Overall time spent
• File system usage (?)

Currently I'm using fully-codegenerated kubernetes repo for manual regression tests, something like that:

ernado@nexus:/src/kubernetes$ golangci-lint cache clean
ernado@nexus:/src/kubernetes$ /usr/bin/time --verbose golangci-lint run --timeout 10m --verbose
INFO [config_reader] Config search paths: [./ /src/kubernetes /src /] 
INFO [lintersdb] Active 10 linters: [deadcode errcheck gosimple govet ineffassign staticcheck structcheck typecheck unused varcheck] 
INFO [loader] Go packages loading at mode 575 (exports_file|imports|deps|files|name|types_sizes|compiled_files) took 6.517475408s 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 263.946976ms
# ...

I want to automate this in some way. Probably, some benchmarks?

[jirfag]

Hi, it looks like we can run golangci-lint on the self repo with all linters enabled and parse time and peak memory from logs. Then check that these values are in the specified range for our CI machines.

And do the same nightly with some large repo like k8s.

[ernado]

So, we can split benches in two categories:

1. Microbenchmarks based on go benchmark framework (testing.B)
2. End-to-end benchmarks

I think both should be run twice (version with and without PR changes) to avoid any noise. The (2) is more obvious to interpret, e.g. "Linting duration increased by 10s" or "Peak memory consumption decreased by 50 Mb" but can be heavy and noisy.

Not sure about (1), but they are integrated in go tooling and have much faster feedback cycle.

Also we can automate (2) on big opensource repo, something like make bench that will build and compare HEAD to latest release, displaying formatted changes.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.