Open njern opened 1 day ago
Hey, thank you for opening your first Issue ! 🙂 If you would like to contribute we have a guide for contributors.
Hello,
In some cases using all the cores doesn't provide the best performance, this is not unexpected because the CPU is one of the limits of a system when using parallel processes.
I think it's because we are hitting other limitations than the CPU related to concurrency access (memory, FS, other processes, etc.).
But as you can see the "best" value for the CPU depends on the system, so it's not possible to do something smart on this topic.
cache | time | |
---|---|---|
default | no cache | 23.078625285s |
-j 4 | no cache | 39.784760618s |
-j 6 | no cache | 28.895793532s |
default | cache | 1.026388844s |
-j 4 | cache | 1.047746156s |
-j 6 | cache | 967.3105ms |
I also have 16 cores, but on Linux amd64.
Also, the cache is not a global thing for golangci-lint, I mean it's not "all or nothing" because the cache is by package, and the invalidation of cache is by package too. Due to that, it's not possible, from the POV of golangci-lint, to really know if there is a cache, and these benchmarks should be evaluated carefully.
Welcome
typecheck
section of the FAQ.Description of the problem
I noticed that both when running a "clean" and a "cached"
golangci-lint
, setting the core limit lower (with the-j
option) seems to improve performance, to a point. It is especially noticeable for cached linting runs.I first noticed this in our own internal repository but I have reproduced it using the golangci-lint repo.
All these commands were run on a Macbook Pro M4 Max (where golangci-lint defaults to using 16 cores).
golangci-lint run -v (after golangci-lint cache clean)
``` INFO golangci-lint has version 1.62.0 built with go1.23.3 from 22b58c9 on 2024-11-10T13:53:08Z INFO [config_reader] Config search paths: [./ /Users/njern/github/golangci-lint /Users/njern/github /Users/njern /Users /] INFO [config_reader] Used config file .golangci.yml INFO [lintersdb] Active 36 linters: [bodyclose copyloopvar depguard dogsled dupl errcheck errorlint funlen gocheckcompilerdirectives gochecknoinits goconst gocritic gocyclo godox gofmt goimports goprintffuncname gosec gosimple govet ineffassign intrange lll misspell mnd nakedret noctx nolintlint revive staticcheck stylecheck testifylint unconvert unparam unused whitespace] INFO [loader] Go packages loading at mode 8767 (types_sizes|compiled_files|exports_file|name|deps|files|imports) took 1.96276675s INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 44.92375ms INFO [linters_context/goanalysis] analyzers took 9m25.254223478s with top 10 stages: gocritic: 6m25.524121759s, buildir: 53.260698388s, goimports: 15.812139791s, dupl: 9.018300673s, the_only_name: 7.497279206s, unconvert: 6.270554796s, ctrlflow: 3.958757812s, fact_deprecated: 3.670158088s, printf: 3.645607995s, gosec: 2.984542789s INFO [runner/skip_dirs] Skipped 2 issues from dir internal/go/robustio by pattern internal/go INFO [runner/skip_dirs] Skipped 2 issues from dir internal/go/mmap by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir internal/go/testenv by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir test/testdata_etc/abspath by pattern test/testdata_etc INFO [runner/skip_dirs] Skipped 1 issues from dir test/testdata_etc/unused_exported/lib by pattern test/testdata_etc INFO [runner/skip_dirs] Skipped 112 issues from dir internal/go/cache by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir internal/go/quoted by pattern internal/go INFO [runner] Issues before processing: 510, after processing: 0 INFO [runner] Processors filtering stat (in/out): filename_unadjuster: 510/510, invalid_issue: 510/510, skip_dirs: 510/384, exclude: 384/384, identifier_marker: 384/384, skip_files: 510/510, cgo: 510/510, path_prettifier: 510/510, autogenerated_exclude: 384/384, exclude-rules: 384/68, nolint: 68/0 INFO [runner] processing took 18.164126ms with stages: exclude-rules: 6.558334ms, autogenerated_exclude: 3.532374ms, nolint: 2.844458ms, identifier_marker: 2.691625ms, path_prettifier: 1.879708ms, skip_dirs: 602.499µs, invalid_issue: 23.958µs, cgo: 18.667µs, filename_unadjuster: 10.499µs, max_same_issues: 542ns, fixer: 292ns, skip_files: 209ns, uniq_by_line: 209ns, exclude: 209ns, sort_results: 167ns, max_from_linter: 125ns, max_per_file_from_linter: 84ns, source_code: 83ns, path_shortener: 42ns, diff: 42ns, path_prefixer: 0s, severity-rules: 0s INFO [runner] linters took 30.540269417s with stages: goanalysis_metalinter: 30.522071875s INFO File cache stats: 426 entries of total size 917.5KiB INFO Memory: 325 samples, avg is 2235.9MB, max is 3410.6MB INFO Execution took 32.555855417s ```golangci-lint run -v -j 4 (after golangci-lint cache clean)
``` INFO golangci-lint has version 1.62.0 built with go1.23.3 from 22b58c9 on 2024-11-10T13:53:08Z INFO [config_reader] Config search paths: [./ /Users/njern/github/golangci-lint /Users/njern/github /Users/njern /Users /] INFO [config_reader] Used config file .golangci.yml INFO [lintersdb] Active 36 linters: [bodyclose copyloopvar depguard dogsled dupl errcheck errorlint funlen gocheckcompilerdirectives gochecknoinits goconst gocritic gocyclo godox gofmt goimports goprintffuncname gosec gosimple govet ineffassign intrange lll misspell mnd nakedret noctx nolintlint revive staticcheck stylecheck testifylint unconvert unparam unused whitespace] INFO [loader] Go packages loading at mode 8767 (files|imports|name|compiled_files|deps|exports_file|types_sizes) took 1.411180584s INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 9.101208ms INFO [linters_context/goanalysis] analyzers took 2m47.626802207s with top 10 stages: gocritic: 1m28.24865404s, buildir: 18.871015413s, goimports: 11.128155453s, dupl: 7.046515498s, the_only_name: 6.109316217s, unconvert: 5.979380999s, fact_deprecated: 1.10383392s, printf: 1.040706755s, inspect: 946.219155ms, gofmt: 928.399373ms INFO [runner/skip_dirs] Skipped 112 issues from dir internal/go/cache by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir internal/go/quoted by pattern internal/go INFO [runner/skip_dirs] Skipped 2 issues from dir internal/go/robustio by pattern internal/go INFO [runner/skip_dirs] Skipped 2 issues from dir internal/go/mmap by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir internal/go/testenv by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir test/testdata_etc/abspath by pattern test/testdata_etc INFO [runner/skip_dirs] Skipped 1 issues from dir test/testdata_etc/unused_exported/lib by pattern test/testdata_etc INFO [runner] Issues before processing: 510, after processing: 0 INFO [runner] Processors filtering stat (in/out): filename_unadjuster: 510/510, path_prettifier: 510/510, exclude-rules: 384/68, nolint: 68/0, autogenerated_exclude: 384/384, identifier_marker: 384/384, exclude: 384/384, cgo: 510/510, invalid_issue: 510/510, skip_files: 510/510, skip_dirs: 510/384 INFO [runner] processing took 16.580957ms with stages: exclude-rules: 6.523333ms, nolint: 2.763708ms, identifier_marker: 2.504375ms, autogenerated_exclude: 2.314209ms, path_prettifier: 1.828168ms, skip_dirs: 592.249µs, invalid_issue: 24.751µs, cgo: 19.208µs, filename_unadjuster: 9.583µs, max_same_issues: 291ns, fixer: 208ns, sort_results: 208ns, skip_files: 125ns, uniq_by_line: 125ns, exclude: 84ns, diff: 83ns, source_code: 83ns, max_from_linter: 82ns, severity-rules: 42ns, path_shortener: 42ns, max_per_file_from_linter: 0s, path_prefixer: 0s INFO [runner] linters took 28.425245709s with stages: goanalysis_metalinter: 28.408623375s INFO File cache stats: 426 entries of total size 917.5KiB INFO Memory: 295 samples, avg is 1534.9MB, max is 2623.2MB INFO Execution took 29.854271375s ```golangci-lint run -v (with cache)
Ran this five times, execution time ranged between 2,14s and 2,37s. ``` INFO golangci-lint has version 1.62.0 built with go1.23.3 from 22b58c9 on 2024-11-10T13:53:08Z INFO [config_reader] Config search paths: [./ /Users/njern/github/golangci-lint /Users/njern/github /Users/njern /Users /] INFO [config_reader] Used config file .golangci.yml INFO [lintersdb] Active 36 linters: [bodyclose copyloopvar depguard dogsled dupl errcheck errorlint funlen gocheckcompilerdirectives gochecknoinits goconst gocritic gocyclo godox gofmt goimports goprintffuncname gosec gosimple govet ineffassign intrange lll misspell mnd nakedret noctx nolintlint revive staticcheck stylecheck testifylint unconvert unparam unused whitespace] INFO [loader] Go packages loading at mode 8767 (files|name|types_sizes|compiled_files|exports_file|deps|imports) took 1.206394916s INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 41.323542ms INFO [linters_context/goanalysis] analyzers took 0s with no stages INFO [runner/skip_dirs] Skipped 3 issues from dir internal/go/quoted by pattern internal/go INFO [runner/skip_dirs] Skipped 1 issues from dir test/testdata_etc/unused_exported/lib by pattern test/testdata_etc INFO [runner/skip_dirs] Skipped 2 issues from dir internal/go/robustio by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir test/testdata_etc/abspath by pattern test/testdata_etc INFO [runner/skip_dirs] Skipped 2 issues from dir internal/go/mmap by pattern internal/go INFO [runner/skip_dirs] Skipped 112 issues from dir internal/go/cache by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir internal/go/testenv by pattern internal/go INFO [runner] Issues before processing: 510, after processing: 0 INFO [runner] Processors filtering stat (in/out): invalid_issue: 510/510, autogenerated_exclude: 384/384, nolint: 68/0, path_prettifier: 510/510, skip_files: 510/510, exclude: 384/384, cgo: 510/510, exclude-rules: 384/68, filename_unadjuster: 510/510, skip_dirs: 510/384, identifier_marker: 384/384 INFO [runner] processing took 16.535252ms with stages: exclude-rules: 6.445667ms, nolint: 2.803958ms, identifier_marker: 2.499625ms, autogenerated_exclude: 2.429708ms, path_prettifier: 1.753499ms, skip_dirs: 551.959µs, invalid_issue: 24.667µs, cgo: 18.334µs, filename_unadjuster: 5.459µs, max_same_issues: 750ns, fixer: 292ns, sort_results: 209ns, diff: 209ns, skip_files: 208ns, max_from_linter: 167ns, exclude: 166ns, uniq_by_line: 125ns, source_code: 125ns, path_shortener: 83ns, severity-rules: 42ns, max_per_file_from_linter: 0s, path_prefixer: 0s INFO [runner] linters took 899.719083ms with stages: goanalysis_metalinter: 883.152667ms INFO File cache stats: 0 entries of total size 0B INFO Memory: 23 samples, avg is 54.1MB, max is 93.8MB INFO Execution took 2.155626167s ```golangci-lint run -v -j 4 (with cache)
Also ran this five times, execution time ranged from 1,43s to 1,69s ``` INFO golangci-lint has version 1.62.0 built with go1.23.3 from 22b58c9 on 2024-11-10T13:53:08Z INFO [config_reader] Config search paths: [./ /Users/njern/github/golangci-lint /Users/njern/github /Users/njern /Users /] INFO [config_reader] Used config file .golangci.yml INFO [lintersdb] Active 36 linters: [bodyclose copyloopvar depguard dogsled dupl errcheck errorlint funlen gocheckcompilerdirectives gochecknoinits goconst gocritic gocyclo godox gofmt goimports goprintffuncname gosec gosimple govet ineffassign intrange lll misspell mnd nakedret noctx nolintlint revive staticcheck stylecheck testifylint unconvert unparam unused whitespace] INFO [loader] Go packages loading at mode 8767 (compiled_files|deps|files|types_sizes|exports_file|imports|name) took 1.38926575s INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 9.9845ms INFO [linters_context/goanalysis] analyzers took 0s with no stages INFO [runner/skip_dirs] Skipped 3 issues from dir internal/go/quoted by pattern internal/go INFO [runner/skip_dirs] Skipped 2 issues from dir internal/go/mmap by pattern internal/go INFO [runner/skip_dirs] Skipped 112 issues from dir internal/go/cache by pattern internal/go INFO [runner/skip_dirs] Skipped 1 issues from dir test/testdata_etc/unused_exported/lib by pattern test/testdata_etc INFO [runner/skip_dirs] Skipped 3 issues from dir internal/go/testenv by pattern internal/go INFO [runner/skip_dirs] Skipped 2 issues from dir internal/go/robustio by pattern internal/go INFO [runner/skip_dirs] Skipped 3 issues from dir test/testdata_etc/abspath by pattern test/testdata_etc INFO [runner] Issues before processing: 510, after processing: 0 INFO [runner] Processors filtering stat (in/out): identifier_marker: 384/384, exclude-rules: 384/68, invalid_issue: 510/510, path_prettifier: 510/510, skip_dirs: 510/384, autogenerated_exclude: 384/384, cgo: 510/510, exclude: 384/384, filename_unadjuster: 510/510, skip_files: 510/510, nolint: 68/0 INFO [runner] processing took 16.082209ms with stages: exclude-rules: 6.301083ms, identifier_marker: 2.826209ms, nolint: 2.734626ms, autogenerated_exclude: 1.970082ms, path_prettifier: 1.677959ms, skip_dirs: 533.166µs, cgo: 17.167µs, invalid_issue: 14.917µs, filename_unadjuster: 4.667µs, max_same_issues: 792ns, diff: 249ns, sort_results: 209ns, fixer: 209ns, skip_files: 207ns, max_from_linter: 166ns, uniq_by_line: 125ns, source_code: 84ns, severity-rules: 83ns, path_shortener: 83ns, path_prefixer: 42ns, max_per_file_from_linter: 42ns, exclude: 42ns INFO [runner] linters took 289.921334ms with stages: goanalysis_metalinter: 273.804875ms INFO File cache stats: 0 entries of total size 0B INFO Memory: 18 samples, avg is 38.0MB, max is 100.3MB INFO Execution took 1.69703725s ```Version of golangci-lint
Configuration
Go environment
Validation
Supporter