Megalinter is very slow

Laurensdc commented 5 months ago

A run on our CI pipeline takes 20 - 30 minutes for our medium sized monorepo.

We've already disabled multiple linters.

Total run time is 35m.

nvuillam commented 5 months ago

On which platform do you run MegaLinter ? Do you runner have multiple cores, that can allow multiple linters to be run in parallel ?

Your system performances seems really slow

Laurensdc commented 5 months ago

We're running the job on Google Cloud Compute Engine instances, through GitLab CI runners: e2-custom-16-24576, t2d-standard-16, t2a-standard-16, t2d-standard-32

nvuillam commented 5 months ago

Do you have issues with other CI/CD jobs, especially docker based ones ? I use custom Gitlab CI runners on Ubuntu servers and it works fine, so I' afraid that the source of your issue is missing hard disk or RAM, or misconfiguration of your runners or the servers they are installed on :/

Laurensdc commented 5 months ago

Our other CI/CD jobs all run as expected. The Megalinter jobs have been taking 20 - 30 minutes consistently for weeks. The instances have 16 vCPUs, 64GB of RAM and run on NVME disks. So I doubt it's a hardware issue.

nvuillam commented 5 months ago

I feel your pain... indeed with such hardware it should take 5 mn not 30 :o :o

Please could you share the beginning of the log like the following one, especially the last line with number of cores detected ?

----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
 - Image Creation Date: 2024-03-14T20:25:19Z
 - Image Revision: 8388d4830fa3234a1cd93bedd2c577e0855cd5e8
 - Image Version: beta
----------------------------------------------------------------------------------------------------
The MegaLinter documentation can be found at:
 - https://megalinter.io/beta
----------------------------------------------------------------------------------------------------
MegaLinter initialization (expand for details)
MegaLinter now collects the files to analyse (expand for details)
  Listing all files in directory [/github/workspace], then filter with:
  - File extensions: .bash, .dash, .env, .gradle, .graphql, .groovy, .gvy, .htm, .html, .json, .json5, .jsonc, .ksh, .kt, .kts, .markdown, .md, .mdx, .nf, .proto, .py, .rst, .sh, .smk, .sql, .txt, .xml, .yaml, .yml
  - File names (regex): Containerfile, Dockerfile, Jenkinsfile, Makefile, Snakefile
  - Excluding regex: (\.automation/test|\.automation/generated|\.venv|\.github/workflows|docs/javascripts|docs/overrides|docs/json-schemas|flavors|clj-kondo|TEMPLATES)
  - Excluding .gitignored files [73]: .automation/test/salesforce/bad/.sfdx/sfdx-config.json, .automation/test/salesforce/good/.sfdx/sfdx-config.json, megalinter-reports/megalinter.log, megalinter/__pycache__/Linter.cpython-312.pyc, megalinter/__pycache__/MegaLinter.cpython-312.pyc, megalinter/__pycache__/Reporter.cpython-312.pyc, megalinter/__pycache__/__init__.cpython-312.pyc, megalinter/__pycache__/alpaca.cpython-312.pyc, megalinter/__pycache__/config.cpython-312.pyc, megalinter/__pycache__/constants.cpython-312.pyc,…(full list in DEBUG)
  Kept [892] files on [2168] found files

  +----MATCHING LINTERS-------------------+------------------------------------------------------------------------+----------------+------------+
  | Descriptor | Linter                   | Criteria                                                               | Matching files | Format/Fix |
  +------------+--------------------------+------------------------------------------------------------------------+----------------+------------+
  | PYTHON     | pylint                   | .py                                                                    | 208            | no         |
  | PYTHON     | black                    | .py                                                                    | 208            | yes        |
  | PYTHON     | flake8                   | .py                                                                    | 208            | no         |
  | PYTHON     | isort                    | .py                                                                    | 208            | yes        |
  | PYTHON     | bandit                   | .py                                                                    | 208            | no         |
  | PYTHON     | mypy                     | .py                                                                    | 208            | no         |
  | PYTHON     | pyright                  | .py                                                                    | 208            | no         |
  | PYTHON     | ruff                     | .py                                                                    | 208            | yes        |
  | BASH       | bash-exec                | .sh|.bash|.dash|.ksh                                                   | 5              | no         |
  | BASH       | shellcheck               | .sh|.bash|.dash|.ksh                                                   | 5              | no         |
  | BASH       | shfmt                    | .sh|.bash|.dash|.ksh                                                   | 5              | yes        |
  | COPYPASTE  | jscpd                    |                                                                        | project        | no         |
  | DOCKERFILE | hadolint                 | Containerfile|Dockerfile                                               | 127            | no         |
  | JSON       | jsonlint                 | .json                                                                  | 19             | no         |
  | JSON       | eslint-plugin-jsonc      | .json|.json5|.jsonc                                                    | 21             | yes        |
  | JSON       | v8r                      | .json                                                                  | 21             | no         |
  | MAKEFILE   | checkmake                | Makefile                                                               | 1              | no         |
  | MARKDOWN   | markdownlint             | .md                                                                    | 261            | yes        |
  | MARKDOWN   | markdown-link-check      | .md                                                                    | 261            | no         |
  | MARKDOWN   | markdown-table-formatter | .md                                                                    | 261            | yes        |
  | OPENAPI    | spectral                 | .yml|.yaml|.json                                                       | 1              | no         |
  | REPOSITORY | checkov                  |                                                                        | project        | no         |
  | REPOSITORY | git_diff                 |                                                                        | project        | no         |
  | REPOSITORY | grype                    |                                                                        | project        | no         |
  | REPOSITORY | secretlint               |                                                                        | project        | no         |
  | REPOSITORY | trivy                    |                                                                        | project        | no         |
  | REPOSITORY | trivy-sbom               |                                                                        | project        | no         |
  | REPOSITORY | trufflehog               |                                                                        | project        | no         |
  | SPELL      | cspell                   |                                                                        | 682            | no         |
  | SPELL      | lychee                   | .md|.mdx|.markdown|.html|.htm|.rst|.txt|.json|.jsonc|.json5|.yaml|.yml | 3[41](https://github.com/oxsecurity/megalinter/actions/runs/8288438431/job/22682948251#step:4:42)            | no         |
  | XML        | xmllint                  | .xml                                                                   | 3              | yes        |
  | YAML       | prettier                 | .yml|.yaml                                                             | 160            | yes        |
  | YAML       | yamllint                 | .yml|.yaml                                                             | 161            | no         |
  | YAML       | v8r                      | .yml|.yaml                                                             | [102](https://github.com/oxsecurity/megalinter/actions/runs/8288438431/job/22682948251#step:4:104)            | no         |
  +------------+--------------------------+------------------------------------------------------------------------+----------------+------------+
Processing linters on [4] parallel cores…

Laurensdc commented 5 months ago

Definitely! Thanks 🙂

----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
 - Image Creation Date: 2023-11-19T16:08:44Z
 - Image Revision: 5364b811bbe86576d832227be174ebc2aa0f5f49
 - Image Version: v7.6.0
----------------------------------------------------------------------------------------------------
The MegaLinter documentation can be found at:
 - https://megalinter.io/7.6.0
----------------------------------------------------------------------------------------------------
MegaLinter initialization (expand for details)
MegaLinter will analyze workspace [/builds/********]
[Activation] ACTION_ACTIONLINT has been set inactive, as subdirectory has not been found: .github/workflows (set value "any" to always activate)
[Activation] ANSIBLE_ANSIBLE_LINT has been set inactive, as subdirectory has not been found: ansible (set value "any" to always activate)
[Activation] EDITORCONFIG_EDITORCONFIG_CHECKER has been set inactive, as none of these files has been found: ['.editorconfig']
[Activation] KUBERNETES_KUBECONFORM has been set inactive, as subdirectory has not been found: kubernetes (set value "any" to always activate)
[Activation] KUBERNETES_HELM has been set inactive, as none of these files has been found: ['Chart.yml', 'Chart.yaml']
[Activation] KUBERNETES_KUBESCAPE has been set inactive, as none of these files has been found: ['Chart.yml', 'Chart.yaml']
[SemgrepLinter] Deactivated because no ruleset has been defined
[Activation] SALESFORCE_SFDX_SCANNER_APEX has been set inactive, as subdirectory has not been found: force-app (set value "any" to always activate)
[Activation] SALESFORCE_SFDX_SCANNER_AURA has been set inactive, as subdirectory has not been found: force-app (set value "any" to always activate)
[Activation] SALESFORCE_SFDX_SCANNER_LWC has been set inactive, as subdirectory has not been found: force-app (set value "any" to always activate)
[Activation] SALESFORCE_LIGHTNING_FLOW_SCANNER has been set inactive, as subdirectory has not been found: force-app (set value "any" to always activate)
[Activation] SPELL_PROSELINT has been set inactive, as none of these files has been found: ['.proselintrc', 'proselint/config.json']
[Activation] SPELL_VALE has been set inactive, as none of these files has been found: ['.vale.ini']
[Activation] SQL_SQLFLUFF has been set inactive, as none of these files has been found: ['.sqlfluff']
[Activation] SWIFT_SWIFTLINT has been set inactive, as none of these files has been found: ['.swiftlint.yml']
MARKDOWN_REMARK_LINT has been temporary disabled in MegaLinter, please use a previous MegaLinter version or wait for the next one !
Skipped linters: ACTION_ACTIONLINT, ANSIBLE_ANSIBLE_LINT, COPYPASTE_JSCPD, CSS_STYLELINT, EDITORCONFIG_EDITORCONFIG_CHECKER, JAVASCRIPT_ES, JAVASCRIPT_PRETTIER, JAVASCRIPT_STANDARD, KUBERNETES_HELM, KUBERNETES_KUBECONFORM, KUBERNETES_KUBESCAPE, MARKDOWN_REMARK_LINT, PYTHON_PYRIGHT, REPOSITORY_KICS, REPOSITORY_SEMGREP, SALESFORCE_LIGHTNING_FLOW_SCANNER, SALESFORCE_SFDX_SCANNER_APEX, SALESFORCE_SFDX_SCANNER_AURA, SALESFORCE_SFDX_SCANNER_LWC, SPELL_PROSELINT, SPELL_VALE, SQL_SQLFLUFF, SWIFT_SWIFTLINT, TYPESCRIPT_ES, TYPESCRIPT_PRETTIER, TYPESCRIPT_STANDARD
To receive reports as email, please set variable EMAIL_REPORTER_EMAIL
MegaLinter now collects the files to analyse (expand for details)
Listing all files in directory [/builds/********], then filter with:
- File extensions: , .R, .RMD, .Rmd, .bash, .bicep, .c, .c++, .cc, .cdxml, .clj, .cljc, .cljs, .coffee, .cpp, .cs, .cu, .cuh, .cxx, .dart, .dash, .edn, .env, .feature, .go, .gradle, .graphql, .groovy, .gvy, .h, .h++, .hcl, .hh, .hpp, .htm, .html, .hxx, .java, .json, .json5, .jsonc, .jsx, .ksh, .kt, .kts, .lua, .markdown, .md, .mdx, .nf, .p6, .php, .pl, .pl6, .pm, .pm6, .pp, .proto, .ps1, .ps1xml, .psd1, .psm1, .psrc, .pssc, .py, .r, .raku, .rakumod, .rakutest, .rb, .rs, .rst, .saas, .scala, .scss, .sh, .smk, .sql, .t, .tex, .tf, .tsx, .txt, .vb, .xml, .yaml, .yml
- File names (regex): Dockerfile, Jenkinsfile, Makefile, Snakefile, package\.json
- Excluding .gitignored files [89313]: .npm/_cacache/content-v2/sha1/ae/47/c1b48545cc2da33df3f42b4ce3399094197e, .npm/_cacache/content-v2/sha512/00/06/e69c41c342fe8249fad6039c17b251adccb3ed02d1d995e28e83175062fc7fb3be8593a4fa655b7bc3620657d8f552b15e0687134d0c5a5c6da846b0e160, .npm/_cacache/content-v2/sha512/00/0d/d3563fb40368ae2284245842bfb6a16306ada3fba3cee98d3325cbf32c016110520edc72f4be5b3d8562e77196c001b2b499aafba19e15d3bf48fea3ccc6, .npm/_cacache/content-v2/sha512/00/46/311fdde31853e7fdada2540c16f3b56e508911d45554281efb370305ee70530e40ebad3fc7a6dfc8ac2274417856dbb8d304371fe5963bc3a462a93330d9, .npm/_cacache/content-v2/sha512/00/5d/ebecfe5d5b12fc331c884d132539140d68e036224005693af893b054ba68cfb51a460d36699743dbd5708ee89783081769d76e8282cf6c331a928e063246, .npm/_cacache/content-v2/sha512/00/aa/5a6251e7f2de1255b3870b2f9be7e28a82f478bebb03f2f6efadb890269b3b7ca0d3923903af2ea38b4ad42630b49336cd78f2f0cf1abc8b2a68e35a9e58, .npm/_cacache/content-v2/sha512/00/e2/2049009ea62258c0fdc04671b1fb95674eed870587736c63f8e5e2f0d6faf7cc1def64b7b279dd6c0bd8676dc39cf7f4ab33233944f42b906cf8692f59a3, .npm/_cacache/content-v2/sha512/00/e8/41c722d576939a7774ee8ed0bd90bffdaf71e74e4c704398203ac62bbef64fd5c085bf31d625de57c590c5b939fe20df2f90a486e52109adddbbbc249b78, .npm/_cacache/content-v2/sha512/00/f3/0111a9e3c9b68fcd4adfa94d08314e48d00b3028c1fb93c3932ecd2fbd5e0a669131913918f3b9d1ff3a5bb933e3fa0011a29eda1946e5d75a670df5bac6, .npm/_cacache/content-v2/sha512/00/f3/341add4aa17073d2222491d07f91fbc63b00b852e245786fdcfb257bc1273babde9c1bf909d078980a7ce2bc3f52079ddc2d5d4c94714d7ceb8f69a7584b,…(full list in DEBUG)
Kept [219] files on [4661] found files
+----MATCHING LINTERS-------------------+------------------------------------------------------------------------+----------------+------------+
| Descriptor | Linter                   | Criteria                                                               | Matching files | Format/Fix |
+------------+--------------------------+------------------------------------------------------------------------+----------------+------------+
| BASH       | bash-exec                | .sh|.bash|.dash|.ksh                                                   | 2              | no         |
| BASH       | shellcheck               | .sh|.bash|.dash|.ksh                                                   | 2              | no         |
| BASH       | shfmt                    | .sh|.bash|.dash|.ksh                                                   | 2              | no         |
| DOCKERFILE | hadolint                 | Dockerfile                                                             | 3              | no         |
| JSON       | jsonlint                 | .json                                                                  | 57             | no         |
| JSON       | eslint-plugin-jsonc      | .json|.json5|.jsonc                                                    | 57             | no         |
| JSON       | v8r                      | .json                                                                  | 57             | no         |
| JSON       | prettier                 | .json                                                                  | 57             | no         |
| JSON       | npm-package-json-lint    | .json|package\.json                                                    | project        | no         |
| MARKDOWN   | markdownlint             | .md                                                                    | 13             | no         |
| MARKDOWN   | markdown-link-check      | .md                                                                    | 13             | no         |
| MARKDOWN   | markdown-table-formatter | .md                                                                    | 13             | no         |
| OPENAPI    | spectral                 | .yml|.yaml|.json                                                       | 5              | no         |
| REPOSITORY | checkov                  |                                                                        | project        | no         |
| REPOSITORY | devskim                  |                                                                        | project        | no         |
| REPOSITORY | dustilock                |                                                                        | project        | no         |
| REPOSITORY | git_diff                 |                                                                        | project        | no         |
| REPOSITORY | gitleaks                 |                                                                        | project        | no         |
| REPOSITORY | grype                    |                                                                        | project        | no         |
| REPOSITORY | secretlint               |                                                                        | project        | no         |
| REPOSITORY | syft                     |                                                                        | project        | no         |
| REPOSITORY | trivy                    |                                                                        | project        | no         |
| REPOSITORY | trivy-sbom               |                                                                        | project        | no         |
| REPOSITORY | trufflehog               |                                                                        | project        | no         |
| SPELL      | cspell                   |                                                                        | 219            | no         |
| SPELL      | lychee                   | .md|.mdx|.markdown|.html|.htm|.rst|.txt|.json|.jsonc|.json5|.yaml|.yml | 83             | no         |
| TSX        | eslint                   | .tsx                                                                   | 115            | no         |
| YAML       | prettier                 | .yml|.yaml                                                             | 13             | no         |
| YAML       | yamllint                 | .yml|.yaml                                                             | 13             | no         |
| YAML       | v8r                      | .yml|.yaml                                                             | 13             | no         |
+------------+--------------------------+------------------------------------------------------------------------+----------------+------------+
Processing linters on [16] parallel cores…

nvuillam commented 5 months ago

Ok so the 16 cores... I'm not a hardware master at all, but what is running too many stuff in parallel would generate some race conditions between the parallel processes... that are all calling git for example ^^

I'll make a PR to allow to force a maximum number of cores by defining a variable, and you'll be able to test with a beta version :)

nvuillam commented 5 months ago

I added the new option in https://github.com/oxsecurity/megalinter/pull/3428 :)

@Laurensdc please try you try again with beta version, and with PARALLEL_PROCESS_NUMBER: 4 in your .mega-linter.yml config file ? :)

Laurensdc commented 5 months ago

Thanks a lot @nvuillam! Our job definitely is faster, but it still takes a while. Is this duration normal?

+----SUMMARY----+--------------------------+---------------+-------+-------+--------+--------------+
| Descriptor    | Linter                   | Mode          | Files | Fixed | Errors | Elapsed time |
+---------------+--------------------------+---------------+-------+-------+--------+--------------+
| ✅ BASH       | bash-exec                | file          |     2 |       |      0 |        0.02s |
| ✅ BASH       | shellcheck               | list_of_files |     2 |       |      0 |        0.89s |
| ✅ BASH       | shfmt                    | list_of_files |     2 |       |      0 |        0.21s |
| ✅ DOCKERFILE | hadolint                 | list_of_files |     3 |       |      0 |        0.92s |
| ✅ JSON       | eslint-plugin-jsonc      | list_of_files |    57 |       |      0 |       50.29s |
| ❌ JSON       | jsonlint                 | list_of_files |    57 |       |      1 |        2.25s |
| ✅ JSON       | npm-package-json-lint    | project       |   n/a |       |      0 |        6.99s |
| ⚠️ JSON       | prettier                 | list_of_files |    57 |       |      1 |        9.72s |
| ✅ JSON       | v8r                      | list_of_files |    57 |       |      0 |      253.79s |
| ⚠️ MARKDOWN   | markdownlint             | list_of_files |    13 |       |     97 |        7.05s |
| ✅ MARKDOWN   | markdown-link-check      | list_of_files |    13 |       |      0 |       34.21s |
| ✅ MARKDOWN   | markdown-table-formatter | list_of_files |    13 |       |      0 |        1.52s |
| ✅ OPENAPI    | spectral                 | file          |     5 |       |      0 |       46.31s |
| ❌ REPOSITORY | checkov                  | project       |   n/a |       |      4 |       67.72s |
| ⚠️ REPOSITORY | devskim                  | project       |   n/a |       |   2581 |       838.7s |
| ✅ REPOSITORY | dustilock                | project       |   n/a |       |      0 |       17.98s |
| ❌ REPOSITORY | gitleaks                 | project       |   n/a |       |      2 |        1.71s |
| ✅ REPOSITORY | git_diff                 | project       |   n/a |       |      0 |        1.21s |
| ❌ REPOSITORY | grype                    | project       |   n/a |       |     80 |       85.98s |
| ✅ REPOSITORY | secretlint               | project       |   n/a |       |      0 |      267.48s |
| ✅ REPOSITORY | syft                     | project       |   n/a |       |      0 |      188.98s |
| ❌ REPOSITORY | trivy                    | project       |   n/a |       |      5 |       51.86s |
| ✅ REPOSITORY | trivy-sbom               | project       |   n/a |       |      0 |       48.07s |
| ✅ REPOSITORY | trufflehog               | project       |   n/a |       |      0 |      308.19s |
| ⚠️ SPELL      | cspell                   | list_of_files |   220 |       |    276 |       26.65s |
| ❌ SPELL      | lychee                   | list_of_files |    83 |       |     10 |        27.6s |
| ✅ TSX        | eslint                   | list_of_files |   115 |       |      0 |       68.38s |
| ⚠️ YAML       | prettier                 | list_of_files |    13 |       |      1 |         7.2s |
| ✅ YAML       | v8r                      | list_of_files |    13 |       |      0 |       37.25s |
| ❌ YAML       | yamllint                 | list_of_files |    13 |       |      5 |        4.89s |
+---------------+--------------------------+---------------+-------+-------+--------+--------------+

Total run time: 19m28s

nvuillam commented 5 months ago

That's much better :)

MegaLinter is a tool but you own the strategy to use it :)

Now is the time to get rid of linters you probably don't need , or exclude some files/folders that do not require to be checked ^^

devskim (slow and does not replies interesting stuff that checkov or trivy will return)
secretlint (not performant, and usually doubling with gitleaks)
trufflehog gathers everything looking like a token / credential and calls the credential provider to check if it is valid. If you don't care to check if they are valid, you can disable the linter as gitleaks will detect anyway
trivy-sbom & syft: if you don't care to generate the SBOM of your project
json_v8r / yaml_v8r checks that your json/yaml files are matching a schema on schemastore.com , and if so if it is compliant. Do you really need it ? Or maybe you could exclude some folders ?

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

oxsecurity / megalinter

Megalinter is very slow #3422