search

oxsecurity / megalinter

🦙 MegaLinter analyzes 50 languages, 22 formats, 21 tooling formats, excessive copy-pastes, spelling mistakes and security issues in your repository sources with a GitHub Action, other CI tools or locally.
https://megalinter.io
GNU Affero General Public License v3.0
1.81k stars 215 forks source link

.secretlintignore doesn't load correctly #2992

Open Sam-Lin-MillersLab opened 9 months ago

Sam-Lin-MillersLab commented 9 months ago

Describe the bug I defined a .secretlintignore file in .github/linter but it doesn't work. If I move it to the root folder, it works.

❌ Linted [REPOSITORY] files with [secretlint]: Found 1 error(s) - (17.52s) (expand for details)
  - Using [secretlint v7.0.3] https://megalinter.io/7.2.0/descriptors/repository_secretlint
  - MegaLinter key: [REPOSITORY_SECRETLINT]
  - Rules config: [/github/workspace/.github/linters/.secretlintrc.json]
  - Ignore file: [/github/workspace/.github/linters/.secretlintignore]
  --Error detail:

To Reproduce Steps to reproduce the behavior:

  1. Go to '…'
  2. Click on '….'
  3. Scroll down to '….'
  4. See error

Expected behavior A clear and concise description of what you expected to happen.

It should load the ignore file from .github/linters/.secretlintignore

Screenshots If applicable, add screenshots to help explain your problem.

Additional context

i am test it on oxsecurity/megalinter/flavors/javascript@v7.4.0

echoix commented 9 months ago

I'm missing a little bit of info to be able to help correctly. What's sure, is that by default, Megalinter shouldn't magically find other configuration files in non-default locations, without it being configured for.

The default location for secretlint's ignore file seems to be at the root of the repo (by https://github.com/secretlint/secretlint#usage, also in https://megalinter.io/latest/descriptors/repository_secretlint/#help-content, and https://github.com/secretlint/secretlint/blob/master/docs/configuration.md)

To configure the tool within Megalinter's config, take a look at the REPOSITORY_SECRETLINT_CONFIG_FILE variable.

At a more global level, LINTER_RULES_PATH, shown here, https://megalinter.io/latest/configuration/#common-variables could be of interest. But since the rules file of secretlint should be a .secretlintrc.json, .secretlintrc.yml, .secretlintrc.yaml, or .secretlintrc.js, maybe that the linter-specific REPOSITORY_SECRETLINT_RULES_PATH is the linter-specific equivalent of LINTER_RULES_PATH, and thus did not apply to REPOSITORY_SECRETLINT_CONFIG_FILE.

Sam-Lin-MillersLab commented 9 months ago

Thanks for the response. This one is not a config file but a ignore file, not sure if that makes any difference.

echoix commented 9 months ago

Do you want us to check at your .mega-linter.yml file?

Is this simply a configuration "error", or a real bug on our side?

The quick fix for this specific linter is to specify REPOSITORY_SECRETLINT_CONFIG_FILE with the location of your ignore file since it is not at the linter's default location (the root of the project). The dirty way (not what I would recommend), is to use the REPOSITORY_SECRETLINT_ARGUMENTS to add arguments to the command called.

Sam-Lin-MillersLab commented 9 months ago

REPOSITORY_SECRETLINT_CONFIG_FILE doesn't work. It somehow caused config not found issue.

I don't think it is my .mega-linter.yml issue, because I have other configs in .github/linters work just fine, and it didn't complain about config not found either

and I don't have anything related to SECRETLINT in mega-linter.yml

❌ Linted [REPOSITORY] files with [secretlint]: Found 1 error(s) - (0.62s) (expand for details)

echoix commented 9 months ago

Since I can't look at your .mega-linter.yml file to answer for myself, what "other configs in .github/linters work just fine" are you talking about, so I can compare what worked and what not?

echoix commented 9 months ago

Ignore file: [/github/workspace/.secretlintignore] --Error detail: Error: secretlint config is not found

This is expected, it states that it looks for an ignore file at /github/workspace/.secretlintignore, but your file isn't there, as per your issue. However, I can't make any further assumptions appart from reading it back, since you didn't mention repro steps, or what config was used to get this error.

Sam-Lin-MillersLab commented 9 months ago

here is the config. I just think it is not the cause.

I have .hadolint.yaml, .jscpd.json, (I changed the threshold and it works), 

# Configuration file for MegaLinter
# See all available variables at https://megalinter.io/configuration/ and in linters documentation

APPLY_FIXES: all
LINTER_RULES_PATH: ".github/linters"
# ENABLE: # If you use ENABLE variable, all other languages/formats/tooling formats will be disabled by default
# ENABLE_LINTERS: # If you use ENABLE_LINTERS variable, all other linters will be disabled by default
DISABLE:
  - CLOUDFORMATION
  - COPYPASTE # Uncomment to disable checks of excessive copy-pastes
  - CSS
  - EDITORCONFIG
  # - SPELL # Uncomment to disable checks of spelling mistakes
  - JAVASCRIPT
  - TEKTON
DISABLE_LINTERS:
  - JSON_PRETTIER
  - CSHARP_DOTNET_FORMAT
  - CSHARP_CSHARPIER
  - JSON_ESLINT_PLUGIN_JSONC
  - REPOSITORY_KICS
  - SPELL_PROSELINT
  - SPELL_CSPELL
  - SPELL_MISSPELL
  - SPELL_LYCHEE
  - SPELL_VALE
  - REPOSITORY_TRUFFLEHOG
  - REPOSITORY_TRIVY
  - REPOSITORY_DEVSKIM
  - REPOSITORY_TRIVY_SBOM
  - REPOSITORY_CHECKOV
  - REPOSITORY_GITLEAKS
  - CSHARP_DOTNET_FORMAT
  - YAML_V8R
DISABLE_ERRORS_LINTERS:
  - PYTHON_BANDIT
  - PYTHON_PYRIGHT
  - REPOSITORY_GRYPE
  - REPOSITORY_SEMGREP
  - SPELL_VALE
  - COPYPASTE_JSCPD
REPOSITORY_DEVSKIM_ARGUMENTS: ["--ignore-globs", "**/megalinter-reports/**,**/.git/**,**/bin/**"]
PRINT_ALL_FILES: false
REPOSITORY_GITLEAKS_PR_COMMITS_SCAN: true
JSON_ESLINT_PLUGIN_JSONC_FILE_NAME: .eslintrc.json
YAML_V8R_FILTER_REGEX_EXCLUDE: '(descriptors|templates/\.mega-linter\.yml|\.codecov\.yml)'
BASH_FILTER_REGEX_EXCLUDE: "(lib)"
MARKDOWN_FILTER_REGEX_EXCLUDE: '(license\.md|docs/licenses)'
MARKDOWN_MARKDOWN_LINK_CHECK_DISABLE_ERRORS: true
SPELL_VALE_FILE_EXTENSIONS:
  - .md
SPELL_FILTER_REGEX_EXCLUDE: (valestyles|\.github|docs/licenses)
DOCKERFILE_HADOLINT_ARGUMENTS: "--ignore DL3003 --ignore DL3007 --ignore DL3013 --ignore DL3016 --ignore DL3018 --ignore DL3028 --ignore DL3059 --ignore DL4001 --ignore DL4006 --ignore SC2015 --ignore SC2016 --ignore SC2039 --ignore SC2086 --ignore SC1091 --ignore SC3046"
REPOSITORY_TRIVY_ARGUMENTS:
  - "--skip-dirs"
  - ".automation/test"
  - "--skip-dirs"
  - ".venv"
SHOW_ELAPSED_TIME: true
FLAVOR_SUGGESTIONS: true
EMAIL_REPORTER: false
FILEIO_REPORTER: false
JSON_REPORTER: true
GITHUB_STATUS_REPORTER: true
VALIDATE_ALL_CODEBASE: false
PLUGINS:
  - https://raw.githubusercontent.com/oxsecurity/megalinter/main/.automation/test/mega-linter-plugin-test/test.megalinter-descriptor.yml
PRE_COMMANDS:
  - command: echo "This is MegaLinter PRE_COMMAND on own MegaLinter ! :)"
    cwd: "root"
POST_COMMANDS:
  - command: echo "This is MegaLinter POST_COMMAND on own MegaLinter ! :)"
    cwd: "workspace"
REPORT_OUTPUT_FOLDER: ".github/megalinter-reports"
echoix commented 9 months ago

What I want to make sure, is that the other linters that are configured and worked are linters that use a single file, and that these files are subject to the global LINTER_RULES_PATH, whilst the secretlint ignore file isn't, and that your configuration doesn't specify it.

echoix commented 9 months ago

here is the config. I just think it is not the cause.

I have .hadolint.yaml, .jscpd.json, (I changed the threshold and it works), 


# Configuration file for MegaLinter

# See all available variables at https://megalinter.io/configuration/ and in linters documentation

APPLY_FIXES: all

LINTER_RULES_PATH: ".github/linters"

# ENABLE: # If you use ENABLE variable, all other languages/formats/tooling formats will be disabled by default

# ENABLE_LINTERS: # If you use ENABLE_LINTERS variable, all other linters will be disabled by default

DISABLE:

  - CLOUDFORMATION

  - COPYPASTE # Uncomment to disable checks of excessive copy-pastes

  - CSS

  - EDITORCONFIG

  # - SPELL # Uncomment to disable checks of spelling mistakes

  - JAVASCRIPT

  - TEKTON

DISABLE_LINTERS:

  - JSON_PRETTIER

  - CSHARP_DOTNET_FORMAT

  - CSHARP_CSHARPIER

  - JSON_ESLINT_PLUGIN_JSONC

  - REPOSITORY_KICS

  - SPELL_PROSELINT

  - SPELL_CSPELL

  - SPELL_MISSPELL

  - SPELL_LYCHEE

  - SPELL_VALE

  - REPOSITORY_TRUFFLEHOG

  - REPOSITORY_TRIVY

  - REPOSITORY_DEVSKIM

  - REPOSITORY_TRIVY_SBOM

  - REPOSITORY_CHECKOV

  - REPOSITORY_GITLEAKS

  - CSHARP_DOTNET_FORMAT

  - YAML_V8R

DISABLE_ERRORS_LINTERS:

  - PYTHON_BANDIT

  - PYTHON_PYRIGHT

  - REPOSITORY_GRYPE

  - REPOSITORY_SEMGREP

  - SPELL_VALE

  - COPYPASTE_JSCPD

REPOSITORY_DEVSKIM_ARGUMENTS: ["--ignore-globs", "**/megalinter-reports/**,**/.git/**,**/bin/**"]

PRINT_ALL_FILES: false

REPOSITORY_GITLEAKS_PR_COMMITS_SCAN: true

JSON_ESLINT_PLUGIN_JSONC_FILE_NAME: .eslintrc.json

YAML_V8R_FILTER_REGEX_EXCLUDE: '(descriptors|templates/\.mega-linter\.yml|\.codecov\.yml)'

BASH_FILTER_REGEX_EXCLUDE: "(lib)"

MARKDOWN_FILTER_REGEX_EXCLUDE: '(license\.md|docs/licenses)'

MARKDOWN_MARKDOWN_LINK_CHECK_DISABLE_ERRORS: true

SPELL_VALE_FILE_EXTENSIONS:

  - .md

SPELL_FILTER_REGEX_EXCLUDE: (valestyles|\.github|docs/licenses)

DOCKERFILE_HADOLINT_ARGUMENTS: "--ignore DL3003 --ignore DL3007 --ignore DL3013 --ignore DL3016 --ignore DL3018 --ignore DL3028 --ignore DL3059 --ignore DL4001 --ignore DL4006 --ignore SC2015 --ignore SC2016 --ignore SC2039 --ignore SC2086 --ignore SC1091 --ignore SC3046"

REPOSITORY_TRIVY_ARGUMENTS:

  - "--skip-dirs"

  - ".automation/test"

  - "--skip-dirs"

  - ".venv"

SHOW_ELAPSED_TIME: true

FLAVOR_SUGGESTIONS: true

EMAIL_REPORTER: false

FILEIO_REPORTER: false

JSON_REPORTER: true

GITHUB_STATUS_REPORTER: true

VALIDATE_ALL_CODEBASE: false

PLUGINS:

  - https://raw.githubusercontent.com/oxsecurity/megalinter/main/.automation/test/mega-linter-plugin-test/test.megalinter-descriptor.yml

PRE_COMMANDS:

  - command: echo "This is MegaLinter PRE_COMMAND on own MegaLinter ! :)"

    cwd: "root"

POST_COMMANDS:

  - command: echo "This is MegaLinter POST_COMMAND on own MegaLinter ! :)"

    cwd: "workspace"

REPORT_OUTPUT_FOLDER: ".github/megalinter-reports"

Try adding 


REPOSITORY_SECRETLINT_CONFIG_FILE: ".github/linters/.secretlintignore"

to your file

Sam-Lin-MillersLab commented 9 months ago

ok i will test more later tonight. thx

Sam-Lin-MillersLab commented 9 months ago

nope that doesn't work either

image

echoix commented 9 months ago

In that case, remove 

REPOSITORY_SECRETLINT_CONFIG_FILE: ".github/linters/.secretlintignore"

And add either 1:

REPOSITORY_SECRETLINT_ARGUMENTS: "--secretlintignore=.github/linters/.secretlintignore"

Or 2:

REPOSITORY_SECRETLINT_ARGUMENTS: "--secretlintignore=.github/linters/.secretlintignore"

REPOSITORY_SECRETLINT_COMMAND_REMOVE_ARGUMENTS: "--secretlintignore"

Or 3:

REPOSITORY_SECRETLINT_ARGUMENTS: "--secretlintignore=.github/linters/.secretlintignore"

REPOSITORY_SECRETLINT_COMMAND_REMOVE_ARGUMENTS: "--secretlintignore=.secretlintignore"
echoix commented 9 months ago

@nvuillam I had a hard time figuring out how to use the _COMMAND_REMOVE_ARGUMENT variables, I'm still not sure if I have to add just the name before the value, or exactly the text that is sent (like with the equal or space or not).

Sam-Lin-MillersLab commented 9 months ago

option 1: doesn't work, saying secretlintignore load multiple times option 2: load secretlintignore from root option 3: doesn't work. saying something wrong on removing like

Traceback (most recent call last):
  File "<frozen runpy>", line [198](https://github.com/Millers-IT/DevOps/actions/runs/6427999683/job/17454487769#step:4:203), in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/site-packages/megalinter-0.1-py3.11.egg/megalinter/run.py", line 14, in <module>
    linter.run()
  File "/usr/local/lib/python3.11/site-packages/megalinter-0.1-py3.11.egg/megalinter/MegaLinter.py", line 234, in run
    self.process_linters_parallel(active_linters, linters_do_fixes)
  File "/usr/local/lib/python3.11/site-packages/megalinter-0.1-py3.11.egg/megalinter/MegaLinter.py", line 343, in process_linters_parallel
    updated_linters = pool_result.get()
                      ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
ValueError: list.remove(x): x not in list
nvuillam commented 8 months ago

Hmmm it seems indeed messy ^^

Why not just defining .secretlintignore at the root of the repo ? ^^

maxisam commented 8 months ago

that's what I'm doing right now. But it's worth to double check

Kurt-von-Laven commented 8 months ago

Thanks for the response. This one is not a config file but a ignore file, not sure if that makes any difference.

@maxisam, you are correct that Secretlint's ignore file is not its config file. You may find it helpful to be aware of what MegaLinter is doing under the hood when it invokes Secretlint.

github-actions[bot] commented 7 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.