Megalinter v6 freezes in Gitlab CI

[schorlton]

Megalinter v6 is freezing in Gitlab CI and I'm not sure how to debug. v5.17.0 works flawlessly.

Description

Running megalinter in Gitlab on a shared runner using your suggested template. I disabled parallel execution to debug.

lint-megalinter:
  stage: lint
  image: oxsecurity/megalinter:v6.5.0
  rules:
    - if: $CI_MERGE_REQUEST_ID
  script: ["true"]
  variables:
    # All available variables are described in documentation
    # https://megalinter.github.io/configuration/
    DEFAULT_WORKSPACE: $CI_PROJECT_DIR
    PARALLEL: "false"
  artifacts:
    when: always
    paths:
      - megalinter-reports
    expire_in: 1 week
  interruptible: true

It successfully pulls the image and runs until hadolint:

Running with gitlab-runner 15.3.0~beta.42.gdb7789ca (db7789ca)
  on blue-4.shared.runners-manager.gitlab.com/default J2nyww-s
Preparing the "docker+machine" executor 03:46
Using Docker executor with image oxsecurity/megalinter:v6.5.0 ...
Pulling docker image oxsecurity/megalinter:v6.5.0 ...
Using docker image sha256:5edff1422a258f93f1ba19a9c48ef1328ef50cabd87054f69eaf9a1aa10970bd for oxsecurity/megalinter:v6.5.0 with digest oxsecurity/megalinter@sha256:586effe9bec89667601bd2168e5576970a648c79b8b5bd2bf19c4440edfb257d ...
Preparing environment 00:04
Running on runner-j2nyww-s-project-18942007-concurrent-0 via runner-j2nyww-s-shared-1660928858-82342b69...
Getting source from Git repository 00:03
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/schorlton/test/.git/
Created fresh repository.
Checking out 2697be3c as refs/merge-requests/874/head...
Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image sha256:5edff1422a258f93f1ba19a9c48ef1328ef50cabd87054f69eaf9a1aa10970bd for oxsecurity/megalinter:v6.5.0 with digest oxsecurity/megalinter@sha256:586effe9bec89667601bd2168e5576970a648c79b8b5bd2bf19c4440edfb257d ...
Setting git safe.directory default: /github/workspace ...
Setting git safe.directory to /tmp/lint ...
[MegaLinter init] ONE-SHOT RUN
[config] /builds/schorlton/test/.mega-linter.yml + Environment variables
----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
 - Image Creation Date: 2022-08-15T19:43:12Z
 - Image Revision: eca13fcbb
 - Image Version: v6.5.0
----------------------------------------------------------------------------------------------------
The MegaLinter documentation can be found at:
 - https://oxsecurity.github.io/megalinter/latest
----------------------------------------------------------------------------------------------------
MegaLinter initialization (expand for details)
[Activation] EDITORCONFIG_EDITORCONFIG_CHECKER has been set inactive, as none of these files has been found: ['.editorconfig']
[Activation] JAVASCRIPT_ES has been set inactive, as none of these files has been found: ['.eslintrc.json', '.eslintrc.yml', '.eslintrc.yaml', '.eslintrc.js', '.eslintrc.cjs', 'package.json:eslintConfig']
[Activation] JSX_ESLINT has been set inactive, as none of these files has been found: ['.eslintrc.json', '.eslintrc.yml', '.eslintrc.yaml', '.eslintrc.js', '.eslintrc.cjs', 'package.json:eslintConfig']
[Activation] REPOSITORY_GOODCHECK has been set inactive, as none of these files has been found: ['goodcheck.yml']
[Activation] SPELL_PROSELINT has been set inactive, as none of these files has been found: ['.proselintrc', 'proselint/config.json']
[Activation] SQL_SQLFLUFF has been set inactive, as none of these files has been found: ['.sqlfluff']
[Activation] SWIFT_SWIFTLINT has been set inactive, as none of these files has been found: ['.swiftlint.yml']
[Activation] TSX_ESLINT has been set inactive, as none of these files has been found: ['.eslintrc.json', '.eslintrc.yml', '.eslintrc.yaml', '.eslintrc.js', '.eslintrc.cjs', 'package.json:eslintConfig']
[Activation] TYPESCRIPT_ES has been set inactive, as none of these files has been found: ['.eslintrc.json', '.eslintrc.yml', '.eslintrc.yaml', '.eslintrc.js', '.eslintrc.cjs', 'package.json:eslintConfig']
MARKDOWN_REMARK_LINT has been temporary disabled in MegaLinter, please use a previous MegaLinter version or wait for the next one !
SNAKEMAKE_SNAKEFMT has been temporary disabled in MegaLinter, please use a previous MegaLinter version or wait for the next one !
Skipped linters: ACTION_ACTIONLINT, ANSIBLE_ANSIBLE_LINT, EDITORCONFIG_EDITORCONFIG_CHECKER, JAVASCRIPT_ES, JAVASCRIPT_PRETTIER, JSX_ESLINT, KUBERNETES_KUBECONFORM, KUBERNETES_KUBEVAL, MARKDOWN_REMARK_LINT, PYTHON_BANDIT, PYTHON_MYPY, REPOSITORY_CHECKOV, REPOSITORY_DEVSKIM, REPOSITORY_GOODCHECK, REPOSITORY_SECRETLINT, REPOSITORY_SEMGREP, REPOSITORY_TRIVY, SALESFORCE_SFDX_SCANNER_APEX, SALESFORCE_SFDX_SCANNER_AURA, SALESFORCE_SFDX_SCANNER_LWC, SNAKEMAKE_SNAKEFMT, SPELL_CSPELL, SPELL_PROSELINT, SQL_SQLFLUFF, SWIFT_SWIFTLINT, TERRAFORM_KICS, TERRAFORM_TERRASCAN, TSX_ESLINT, TYPESCRIPT_ES, TYPESCRIPT_PRETTIER
To receive reports as email, please set variable EMAIL_REPORTER_EMAIL
MegaLinter now collects the files to analyse (expand for details)
Listing all files in directory [/builds/schorlton/test], then filter with:
- File extensions: , .R, .RMD, .Rmd, .bash, .c, .c++, .cc, .cdxml, .clj, .cljc, .cljs, .coffee, .cpp, .cs, .css, .cu, .cuh, .cxx, .dart, .dash, .edn, .env, .feature, .go, .gradle, .graphql, .groovy, .gvy, .h, .h++, .hcl, .hh, .hpp, .htm, .html, .hxx, .java, .js, .json, .json5, .jsonc, .ksh, .kt, .kts, .lua, .md, .nf, .p6, .php, .pl, .pl6, .pm, .pm6, .pp, .proto, .ps1, .ps1xml, .psd1, .psm1, .psrc, .pssc, .py, .r, .raku, .rakumod, .rakutest, .rb, .rs, .rst, .saas, .scala, .scss, .sh, .smk, .sql, .t, .tex, .tf, .ts, .vb, .xml, .yaml, .yml
- File names (regex): Dockerfile, Jenkinsfile, Makefile, Snakefile
- Excluding regex: (redacted)
- Excluding .gitignored files [1]: /builds/schorlton/test/megalinter-reports/megalinter.log
Kept [297] files on [316] found files
+----MATCHING LINTERS-------------------+--------------------------------------+----------------+------------+
| Descriptor | Linter                   | Criteria                             | Matching files | Format/Fix |
+------------+--------------------------+--------------------------------------+----------------+------------+
| PYTHON     | pylint                   | .py                                  | 84             | no         |
| PYTHON     | black                    | .py                                  | 84             | no         |
| PYTHON     | flake8                   | .py                                  | 84             | no         |
| PYTHON     | isort                    | .py                                  | 84             | no         |
| BASH       | bash-exec                | .sh|.bash|.dash|.ksh                 | 26             | no         |
| BASH       | shellcheck               | .sh|.bash|.dash|.ksh                 | 26             | no         |
| BASH       | shfmt                    | .sh|.bash|.dash|.ksh                 | 26             | no         |
| COPYPASTE  | jscpd                    |                                      | project        | no         |
| DOCKERFILE | hadolint                 | Dockerfile                           | 50             | no         |
| GROOVY     | npm-groovy-lint          | .groovy|.gvy|.gradle|.nf|Jenkinsfile | 55             | no         |
| JSON       | jsonlint                 | .json                                | 7              | no         |
| JSON       | eslint-plugin-jsonc      | .json|.json5|.jsonc                  | 7              | no         |
| JSON       | v8r                      | .json                                | 7              | no         |
| JSON       | prettier                 | .json                                | 7              | no         |
| MARKDOWN   | markdownlint             | .md                                  | 5              | no         |
| MARKDOWN   | markdown-link-check      | .md                                  | 5              | no         |
| MARKDOWN   | markdown-table-formatter | .md                                  | 5              | no         |
| PROTOBUF   | protolint                | .proto                               | 1              | no         |
| REPOSITORY | dustilock                |                                      | project        | no         |
| REPOSITORY | git_diff                 |                                      | project        | no         |
| REPOSITORY | gitleaks                 |                                      | project        | no         |
| REPOSITORY | syft                     |                                      | project        | no         |
| SPELL      | misspell                 |                                      | 297            | no         |
| TERRAFORM  | tflint                   | .tf                                  | 6              | no         |
| TERRAFORM  | terragrunt               | .hcl                                 | 1              | no         |
| TERRAFORM  | terraform-fmt            | .tf                                  | 6              | no         |
| TERRAFORM  | checkov                  | .tf                                  | 6              | no         |
| YAML       | prettier                 | .yml|.yaml                           | 8              | no         |
| YAML       | yamllint                 | .yml|.yaml                           | 8              | no         |
| YAML       | v8r                      | .yml|.yaml                           | 8              | no         |
+------------+--------------------------+--------------------------------------+----------------+------------+
✅ Linted [PYTHON] files with [pylint] successfully - (14.14s) (expand for details)
- Using [pylint v2.14.5] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/python_pylint
- MegaLinter key: [PYTHON_PYLINT]
- Rules config: [.pylintrc]
- Number of files analyzed: [84]
✅ Linted [PYTHON] files with [black] successfully - (3.62s) (expand for details)
- Using [black v22.6.0] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/python_black
- MegaLinter key: [PYTHON_BLACK]
- Rules config: [pyproject.toml]
- Number of files analyzed: [84]
✅ Linted [PYTHON] files with [flake8] successfully - (2.62s) (expand for details)
- Using [flake8 v5.0.4] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/python_flake8
- MegaLinter key: [PYTHON_FLAKE8]
- Rules config: [/builds/schorlton/test/.flake8]
- Number of files analyzed: [84]
✅ Linted [PYTHON] files with [isort] successfully - (0.3s) (expand for details)
- Using [isort v5.10.1] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/python_isort
- MegaLinter key: [PYTHON_ISORT]
- Rules config: [/builds/schorlton/test/.isort.cfg]
- Number of files analyzed: [84]
✅ Linted [BASH] files with [bash-exec] successfully - (0.11s) (expand for details)
- Using [bash-exec v5.1.16] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/bash_bash_exec
- MegaLinter key: [BASH_EXEC]
- Rules config: identified by [bash-exec]
- Number of files analyzed: [26]
✅ Linted [BASH] files with [shellcheck] successfully - (0.57s) (expand for details)
- Using [shellcheck v0.8.0] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/bash_shellcheck
- MegaLinter key: [BASH_SHELLCHECK]
- Rules config: identified by [shellcheck]
- Number of files analyzed: [26]
✅ Linted [BASH] files with [shfmt] successfully - (0.11s) (expand for details)
- Using [shfmt v3.6.0] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/bash_shfmt
- MegaLinter key: [BASH_SHFMT]
- Rules config: identified by [shfmt]
- Number of files analyzed: [26]
✅ Linted [COPYPASTE] files with [jscpd] successfully - (4.07s) (expand for details)
- Using [jscpd v3.3.26] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/copypaste_jscpd
- MegaLinter key: [COPYPASTE_JSCPD]
- Rules config: [.jscpd.json]
✅ Linted [DOCKERFILE] files with [hadolint] successfully - (3.33s) (expand for details)
- Using [hadolint v2.10.0] https://oxsecurity.github.io/megalinter/latest/v6.5.0/descriptors/dockerfile_hadolint
- MegaLinter key: [DOCKERFILE_HADOLINT]
- Rules config: [/builds/schorlton/test/.hadolint.yaml]
- Number of files analyzed: [50]

At this point, it times out after an hour. The linting works flawlessly in v6 locally and is quick. When I follow along locally, it seems that npm-groovy-lint runs after hadolint, so I wonder if this is what is freezing?

Thanks for any tips!

[Kurt-von-Laven]

I would try running with LOG_LEVEL: DEBUG and DISABLE_LINTERS: GROOVY. If you can prove that a particular linter is hanging, then I would try running it directly to assess whether the issue is related to MegaLinter.

[nvuillam]

@schorlton I recently released a new npm-groovy-lint version that can accept a list of files Please could you disable groovy like suggested by @Kurt-von-Laven , so we can know if npm-groovy-lint is responsible for the timeout ?

Or any chance we could have access to the gitlab repo ?

[nvuillam]

(if npm-groovy-lint is guilty, you may also use GROOVY_NPM_GROOVY_LINT_CLI_LINT_MODE: file, it will be slower but may work before a fix is published )

[schorlton]

Thank you both for your feedback. I'm confirming it's npm-groovy-lint, and changing GROOVY_NPM_GROOVY_LINT_CLI_LINT_MODE did not help. The linting works locally which makes me think maybe it's a resource limitation of the GitLab CI environment? It's weird that it broke going from v5 to v6. I unfortunately cannot provide access to our repo, but please let me know if there's any other debugging I can help with. You may be able to reproduce this issue by simply running a big (several hundred line) file through npm-groovy-lint on GitLab CI - in file mode, it looks like it was frozen on the first file.

[nvuillam]

Please can you try defining the following in your MegaLinter config file ?

GROOVY_NPM_GROOVY_LINT_ARGUMENTS: --noserver

[schorlton]

Please can you try defining the following in your MegaLinter config file ? GROOVY_NPM_GROOVY_LINT_ARGUMENTS: --noserver

Should I also keep GROOVY_NPM_GROOVY_LINT_CLI_LINT_MODE: file or remove that before running?

[nvuillam]

you can try first to remove GROOVY_NPM_GROOVY_LINT_CLI_LINT_MODE: file and see what happens with only GROOVY_NPM_GROOVY_LINT_ARGUMENTS: --noserver :)

[schorlton]

you can try first to remove GROOVY_NPM_GROOVY_LINT_CLI_LINT_MODE: file and see what happens with only GROOVY_NPM_GROOVY_LINT_ARGUMENTS: --noserver :)

That fixed it!! :smile:

[schorlton]

Addendum: premature excitement. It got further but freezes on:

npm-groovy-lint results in 128 linted files:
┌─────────┬───────────┬─────────────┐
│ (index) │ Severity  │ Total found │
├─────────┼───────────┼─────────────┤
│    0    │  'Error'  │      0      │
│    1    │ 'Warning' │      7      │
│    2    │  'Info'   │     129     │
└─────────┴───────────┴─────────────┘
Failure: 0 error(s) have been found 
 7 warning(s) have been found 
 129 info(s) have been found
Linter version command: ['/node-deps/node_modules/.bin/npm-groovy-lint', '--version']

[Kurt-von-Laven]

You may want to try running npm-groovy-lint directly in GitLab CI (without MegaLinter). If you can reproduce the hang that way, this probably belongs in npm-groovy-lint's issue tracker. Maybe they need to expose a config option that limits resource consumption in some way?

[nvuillam]

@schorlton Oh, so the --version is boring :o Do you have some public repo where I could see the problem ?

@Kurt-von-Laven "they" is kind of .... me 🤣 I'm the creator/maintainer of npm-groovy-lint... and only maintainer 😭

[Kurt-von-Laven]

Ha ha ha ha, so sorry.

[schorlton]

@Kurt-von-Laven - yes I'm sorry for polluting megalinter repo with likely external issue. Thanks for understanding. @nvuillam - big thanks for your help on debugging. Unfortunately, this is a private repo and I can't share as it's protected IP. My best suggestion is to try running in gitlab CI on ~128 nextflow files with ~129 infos and ~7 warnings...perhaps running on one of the nf-core repos may reproduce?

[nvuillam]

That may indeed be caused by the big amount of nextflow files.... I'll test with npm-groovy-lint and try to pimp it :)

[schorlton]

@nvuillam, wondering if you've been able to reproduce? Would be really nice to switch to Megalinter v6 with all of its improvements, but this is precluding the switch. Thanks!

[nvuillam]

@schorlton I'm finalizing a pull request on codenarc to arrange that, then it will arrive in npm-groovy-lint, then megalinter :) Slow but progressing :)

[nvuillam]

https://github.com/CodeNarc/CodeNarc/pull/717

[schorlton]

Amazing. If I understand, it was a performance issue? Was too many files crashing it on Gitlab due to resource constraints?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[schorlton]

Please stay open :pray:

[nvuillam]

@schorlton I have to wait for CodeNarc new release... they are not frequent as MegaLinter... ^^

https://github.com/CodeNarc/CodeNarc

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[schorlton]

@nvuillam , looks like there's a new CodeNarc release...any update on bringing it over into npm-groovy-lint/here?

[nvuillam]

@nvuillam , looks like there's a new CodeNarc release...any update on bringing it over into npm-groovy-lint/here?

I have a pending branch to upgrade CodeNarc within npm-groovy-lint... but it has issues and I have big difficulties to find time :/

https://github.com/nvuillam/npm-groovy-lint/pull/258/files

if you are interested or know someone you can try to make it work :/

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[schorlton]

It may be planned?

[nvuillam]

Planned yet... but scheduled, it's harder to say, these days are hard to find time, it will probably wait for my next vacations :/

[schorlton]

@nvuillam, the GitLab team (credit to them) has troubleshooted a bit why megalinter v6/7 is freezing in GitLab CI and came up with:

npm-groovy-lint seems to hang when executing ['/node-deps/node_modules/.bin/npm-groovy-lint', '--version']. We look at the code base and pull the container to test around a bit. Kenneth suspects the --codenarcargs might be causing the problem and we try removing line #197, #198 , and #204. It seems to improve the command run time a lot so we tested on the GitLab SaaS runner and the job no longer hang.

The code we change to make is work is in config/gitlab_ci/lint.yaml: lint-megalinter: stage: lint image: name: oxsecurity/megalinter:v7.0.2 entrypoint: ["/bin/sh", "-c", "sed -i '/codeNarcVersionLinter/d' /node-deps/node_modules/npm-groovy-lint/lib/groovy-lint.js; sed -i '/codeNarcVersionLines/d' /node-deps/node_modules/npm-groovy-lint/lib/groovy-lint.js; /entrypoint.sh"] rules: ...

Does this give you any hints why it may be freezing or do you have any ideas?

Thanks!

[nvuillam]

I'll try to have a look and solve the bug on npm-groovy-lint before it becomes an abandonware, many thanks to you and to the gitlab team :)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[sanmai-NL]

Not stale.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[fnickels]

I am seeing a similar issue.
Running npm-groovy-lint and even something as basic as npm-groovy-lint -v, in a container locally runs fine. But running the same container in Jenkins and the app just sits there until Jenkins times out the image or I kill the Job.

13:08:46  + npm-groovy-lint
Aborted by [fnickels](https://rce-controller01.****/user/fnickels)
13:10:19  Sending interrupt signal to process
13:10:19  Terminated
13:10:19  script returned exit code 143

I have attempted to use the --noserver, --verbose, and other flags and get no other indications as to why the app is hanging. I recognize it is probably something about environment, but npm-groovy-lint does not provide any useful diagnostics I am aware of about where and why it is hanging.

Any suggestions about what to try? Here are the details when I run the same container locally:

# npm-groovy-lint -v
GroovyLint: Successfully processed CodeNarc:
CodeNarc version 3.1.0

npm-groovy-lint version 11.1.1

Embeds:
CodeNarc version 3.1.0
- Groovy version 3.0.9 (superlite)
# node --version
v19.7.0
# npm --version
9.5.0
# java --version
openjdk 11.0.17 2022-10-18 LTS
OpenJDK Runtime Environment Corretto-11.0.17.8.1 (build 11.0.17+8-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.17.8.1 (build 11.0.17+8-LTS, mixed mode)

Is there somewhere in the code where the app never returns and endlessly waits for something?

[nvuillam]

npm-groovy-lint is currently a mess because of newest java/groovy/codenarc versions (we don't know which ones yet) , and contrary to MegaLinter, I'm the only maintainer there, and i don't code in java for years.... so I don't succeed to solve it for now ... Some contributors are trying to help, but meanwhile u might just disable the linter :/

[fnickels]

I can appreciate that. I too have not looked at Java in years, but I might take a peek and see what I can find.

[nvuillam]

@fnickels that would be amazing

Current PR is https://github.com/nvuillam/npm-groovy-lint/pull/286

There is also another attempt by @mheiges -> https://github.com/nvuillam/npm-groovy-lint/pull/292

[fnickels]

I made some progress getting things to run by doing the following:

 java --class-path '/usr/local/lib/node_modules/npm-groovy-lint/lib/java/CodeNarcServer.jar:/usr/local/lib/node_modules/npm-groovy-lint/lib/java/*' \
            com.nvuillam.CodeNarcServer --server  &

      sleep 5
      npm-groovy-lint --noserver ./
      jobs
      kill %1
      sleep 5
      jobs

but I can not get npm-groovy-lint -v to work. I briefly looked at the code and realized how rusty my java & node skills are. I will cross-post in the other repo in case this is helpful to others. I will also take a look at the 2 PRs you have pointed out.

[nvuillam]

@fnickels I'm rusty in java & groovy too, that's what I can't succeed to fix npm-groovy-lint for now :/ But some others are trying, let's keep hope :)

[schorlton]

I just tried v7.4.0 in GitLab CI and it worked for me. Really not sure what changed. Will leave this open until I confirm a couple times and then close.

[nvuillam]

@schorlton thanks for the feedback :)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.