IGNORE_GITIGNORED_FILES not applied for cfn-lint

[ashokm]

Describe the bug The documentation states that the IGNORE_GITIGNORED_FILES env var has a default setting of true and that:

If set to true, MegaLinter will skip files ignored by git using .gitignore file

This has been working as expected in v6, but has stopped working when v7 was released. After v7, we see cfn-lint not behaving as expected and it is now including directories that are set in .gitignore.

To Reproduce For a CDK-based project as an example, add cdk.out to .gitignore in the project repository. Run MegaLinter v7 (running v7.3.0 at time of writing). You'll see in the output (in Azure Pipelines in this case) that /tmp/lint/cdk.out/manifest.json, /tmp/lint/cdk.out/tree.json etc are listed as being .gitignored in this line:

- Excluding .gitignored files

However, after that, these files are still included in the linting scan:

...
2023-09-14T07:34:59.9630900Z ##[group]❌ Linted [CLOUDFORMATION] files with [cfn-lint]: Found 1 error(s) - (6.41s) (expand for details)
2023-09-14T07:34:59.9632415Z - Using [cfn-lint v0.79.6] https://megalinter.io/7.3.0/descriptors/cloudformation_cfn_lint
2023-09-14T07:34:59.9633131Z - MegaLinter key: [CLOUDFORMATION_CFN_LINT]
2023-09-14T07:34:59.9635473Z - Rules config: [.cfnlintrc.yml]
2023-09-14T07:34:59.9636044Z - Number of files analyzed: [4]
2023-09-14T07:34:59.9636477Z --Error detail:
2023-09-14T07:34:59.9636810Z E1001 Missing top level template section Resources
2023-09-14T07:34:59.9716728Z cdk.out/manifest.json:1:1
2023-09-14T07:34:59.9717319Z 
2023-09-14T07:34:59.9717740Z E1001 Top level template section version is not valid
2023-09-14T07:34:59.9717985Z cdk.out/manifest.json:2:3
2023-09-14T07:34:59.9718177Z 
2023-09-14T07:34:59.9718350Z E1001 Top level template section artifacts is not valid
2023-09-14T07:34:59.9718554Z cdk.out/manifest.json:3:3
...
2023-09-14T07:34:59.9721633Z E1001 Missing top level template section Resources
2023-09-14T07:34:59.9721814Z cdk.out/tree.json:1:1
2023-09-14T07:34:59.9721883Z 
2023-09-14T07:34:59.9722044Z E1001 Top level template section version is not valid
2023-09-14T07:34:59.9722212Z cdk.out/tree.json:2:3
2023-09-14T07:34:59.9722295Z 
2023-09-14T07:34:59.9722438Z E1001 Top level template section tree is not valid
2023-09-14T07:34:59.9722619Z cdk.out/tree.json:3:3
...

Additional context

• No custom mega-linter.yml configuration file is being used
• No custom .cfnlintrc.yml configuration file is being used

[nvuillam]

@ashokm that's strange, cfn-lint is called with list_of_files so .gitignore files should be indeed ignored (https://megalinter.io/latest/descriptors/cloudformation_cfn_lint/#how-the-linting-is-performed )

Please could you run your workflow with LOG_LEVEL: DEBUG so we can investigate ?

[ashokm]

Any specific part of the output you're interested in? There is a lot of information in the logs, some of which I cannot share for obvious reasons. Below is an extract of some parts of the log with LOG_LEVEL set to 'DEBUG':

...
...
2023-09-19T06:37:37.7853574Z Listing all files in directory [/tmp/lint], then filter with:
2023-09-19T06:37:37.7853761Z Root dir content:
2023-09-19T06:37:37.7854126Z - .checkov.yml
2023-09-19T06:37:37.7854319Z - .gitignore
2023-09-19T06:37:37.7854505Z - .npmrc
2023-09-19T06:37:37.7854823Z - README.md
2023-09-19T06:37:37.7855127Z - azure-pipelines.yml
2023-09-19T06:37:37.7855330Z - cdk.context.json
2023-09-19T06:37:37.7855517Z - cdk.json
2023-09-19T06:37:37.7855705Z - jest.config.js
2023-09-19T06:37:37.7855887Z - package-lock.json
2023-09-19T06:37:37.7856079Z - package.json
2023-09-19T06:37:37.7856268Z - tsconfig.json
2023-09-19T06:37:37.7867150Z All found files before filtering:
2023-09-19T06:37:37.7867425Z - .checkov.yml
2023-09-19T06:37:37.7867643Z - .devcontainer/Dockerfile
2023-09-19T06:37:37.7867879Z - .devcontainer/devcontainer.json
2023-09-19T06:37:37.7868208Z - .gitignore
2023-09-19T06:37:37.7868370Z - .npmrc
2023-09-19T06:37:37.7868561Z - .vscode/settings.json
2023-09-19T06:37:37.7868750Z - README.md
2023-09-19T06:37:37.7868943Z - azure-pipelines.yml
2023-09-19T06:37:37.7869130Z - bin/blah-cdk-stack.d.ts
2023-09-19T06:37:37.7869330Z - bin/blah-cdk-stack.js
2023-09-19T06:37:37.7869527Z - bin/blah-cdk-stack.ts
2023-09-19T06:37:37.7869718Z - cdk.context.json
2023-09-19T06:37:37.7870107Z - cdk.json
2023-09-19T06:37:37.7874031Z - cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v2-handler.js
2023-09-19T06:37:37.7874489Z - cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v3-handler.bundled.js
2023-09-19T06:37:37.7874900Z - cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/index.js
2023-09-19T06:37:37.7875276Z - cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/shared.js
2023-09-19T06:37:37.7875686Z - cdk.out/asset.b992a5b1efd6715c28d74eda79615c6b4c3cf54e6364f2b2166dacf0f6c68602/update-blah-ip-set.py
2023-09-19T06:37:37.7876075Z - cdk.out/cdk.out
2023-09-19T06:37:37.7876288Z - cdk.out/manifest.json
2023-09-19T06:37:37.7876497Z - cdk.out/tree.json
2023-09-19T06:37:37.7876713Z - cdk.out/blah-acl-dev.assets.json
2023-09-19T06:37:37.7876958Z - cdk.out/blah-acl-dev.template.json
2023-09-19T06:37:37.7877176Z - jest.config.js
2023-09-19T06:37:37.7877397Z - lib/blah-cdk-stack-stack.d.ts
2023-09-19T06:37:37.7882210Z - lib/blah-cdk-stack-stack.js
2023-09-19T06:37:37.7882486Z - lib/blah-cdk-stack-stack.ts
2023-09-19T06:37:37.7882690Z - package-lock.json
2023-09-19T06:37:37.7882996Z - package.json
2023-09-19T06:37:37.7883197Z - test/blah-cdk-stack.test.d.ts
2023-09-19T06:37:37.7883421Z - test/blah-cdk-stack.test.js
2023-09-19T06:37:37.7883642Z - test/blah-cdk-stack.test.ts
2023-09-19T06:37:37.7883843Z - tsconfig.json
2023-09-19T06:37:37.7884053Z - File extensions: .json, .yaml, .yml
2023-09-19T06:37:37.7884439Z Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')
2023-09-19T06:37:37.7898372Z Popen(['git', 'ls-files', '--exclude-standard', '--ignored', '--others', '--cached'], cwd=/tmp/lint, universal_newlines=False, shell=None, istream=None)
2023-09-19T06:37:38.0633638Z - Excluding .gitignored files [10772]: /tmp/lint/bin/blah-cdk-stack.d.ts, /tmp/lint/bin/blah-cdk-stack.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v2-handler.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v3-handler.bundled.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/index.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/shared.js, /tmp/lint/cdk.out/asset.b992a5b1efd6715c28d74eda79615c6b4c3cf54e6364f2b2166dacf0f6c68602/update-blah-ip-set.py, /tmp/lint/cdk.out/cdk.out, /tmp/lint/cdk.out/manifest.json, /tmp/lint/cdk.out/tree.json, /tmp/lint/cdk.out/blah-acl-dev.assets.json, /tmp/lint/cdk.out/blah-acl-dev.template.json, /tmp/lint/lib/blah-cdk-stack-stack.d.ts, /tmp/lint/lib/blah-cdk-stack-stack.js, /tmp/lint/node_modules/.bin/acorn, /tmp/lint/node_modules/.bin/browserslist, /tmp/lint/node_modules/.bin/cdk, /tmp/lint/node_modules/.bin/esparse, /tmp/lint/node_modules/.bin/esvalidate, /tmp/lint/node_modules/.bin/import-local-fixture, /tmp/lint/node_modules/.bin/jest, /tmp/lint/node_modules/.bin/js-yaml,
...
...
 /tmp/lint/node_modules/yn/readme.md, /tmp/lint/node_modules/yocto-queue/index.d.ts, /tmp/lint/node_modules/yocto-queue/index.js, /tmp/lint/node_modules/yocto-queue/license, /tmp/lint/node_modules/yocto-queue/package.json, /tmp/lint/node_modules/yocto-queue/readme.md, /tmp/lint/test/blah-cdk-stack.test.d.ts, /tmp/lint/test/blah-cdk-stack.test.js
2023-09-19T06:37:38.1360909Z Kept [13] files on [33] found files
2023-09-19T06:37:38.1361164Z Kept files before applying linter filters:
2023-09-19T06:37:38.1361588Z - .checkov.yml
2023-09-19T06:37:38.1361827Z - .devcontainer/devcontainer.json
2023-09-19T06:37:38.1362059Z - .vscode/settings.json
2023-09-19T06:37:38.1362260Z - azure-pipelines.yml
2023-09-19T06:37:38.1362469Z - cdk.context.json
2023-09-19T06:37:38.1362662Z - cdk.json
2023-09-19T06:37:38.1362867Z - cdk.out/manifest.json
2023-09-19T06:37:38.1363184Z - cdk.out/tree.json
2023-09-19T06:37:38.1363410Z - cdk.out/blah-acl-dev.assets.json
2023-09-19T06:37:38.1363771Z - cdk.out/blah-acl-dev.template.json
2023-09-19T06:37:38.1363979Z - package-lock.json
2023-09-19T06:37:38.1364257Z - package.json
2023-09-19T06:37:38.1364430Z - tsconfig.json
2023-09-19T06:37:38.1365290Z [Filters] {'name': 'CLOUDFORMATION_CFN_LINT', 'filter_regex_include': None, 'filter_regex_exclude_descriptor': None, 'filter_regex_exclude_linter': None, 'files_sub_directory': None, 'lint_all_files': False, 'lint_all_other_linters_files': False, 'file_extensions': ['.yml', '.yaml', '.json'], 'file_names_regex': [], 'file_names_not_ends_with': [], 'file_contains_regex': ['AWSTemplateFormatVersion', '(AWS|Alexa|Custom)::'], 
'file_contains_regex_extensions': []}
2023-09-19T06:37:38.1366255Z CLOUDFORMATION_CFN_LINT linter kept 4 files after applying linter filters:
2023-09-19T06:37:38.1366499Z - cdk.out/manifest.json
2023-09-19T06:37:38.1366672Z - cdk.out/tree.json
2023-09-19T06:37:38.1366875Z - cdk.out/blah-acl-dev.assets.json
2023-09-19T06:37:38.1367092Z - cdk.out/blah-acl-dev.template.json
2023-09-19T06:37:38.1367163Z 
2023-09-19T06:37:38.1367427Z +----MATCHING LINTERS-------+------------------+----------------+------------+
2023-09-19T06:37:38.1367644Z | Descriptor     | Linter   | Criteria         | Matching files | Format/Fix |
2023-09-19T06:37:38.1367945Z +----------------+----------+------------------+----------------+------------+
2023-09-19T06:37:38.1368238Z | CLOUDFORMATION | cfn-lint | .yml|.yaml|.json | 4              | no         |
2023-09-19T06:37:38.1368529Z +----------------+----------+------------------+----------------+------------+
2023-09-19T06:37:38.1368711Z ##[endgroup]
2023-09-19T06:37:38.1369160Z [cfn-lint] command: ['cfn-lint', '--config-file', '/action/lib/.automation/.cfnlintrc.yml', 'cdk.out/manifest.json', 'cdk.out/tree.json', 'cdk.out/blah-acl-dev.assets.json', 'cdk.out/blah-acl-dev.template.json']
2023-09-19T06:37:38.1369457Z [cfn-lint] CWD: /tmp/lint
2023-09-19T06:37:44.2570068Z [cfn-lint] result: 14 E1001 Missing top level template section Resources
2023-09-19T06:37:44.2570803Z cdk.out/manifest.json:1:1
2023-09-19T06:37:44.2572702Z 
2023-09-19T06:37:44.2573125Z E1001 Top level template section version is not valid
2023-09-19T06:37:44.2574459Z cdk.out/manifest.json:2:3
2023-09-19T06:37:44.2575075Z 
2023-09-19T06:37:44.2575491Z E1001 Top level template section artifacts is not valid
2023-09-19T06:37:44.2575886Z cdk.out/manifest.json:3:3
...
...

[nvuillam]

I think this line helps :)

Excluding .gitignored files [10772]: /tmp/lint/bin/blah-cdk-stack.d.ts, /tmp/lint/bin/blah-cdk-stack.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v2-handler.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v3-handler.bundled.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/index.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/shared.js, /tmp/lint/cdk.out/asset.b992a5b1efd6715c28d74eda79615c6b4c3cf54e6364f2b2166dacf0f6c68602/update-blah-ip-set.py, /tmp/lint/cdk.out/cdk.out, /tmp/lint/cdk.out/manifest.json, /tmp/lint/cdk.out/tree.json, /tmp/lint/cdk.out/blah-acl-dev.assets.json, /tmp/lint/cdk.out/blah-acl-dev.template.json, /tmp/lint/lib/blah-cdk-stack-stack.d.ts, /tmp/lint/lib/blah-cdk-stack-stack.js, /tmp/lint/node_modules/.bin/acorn, /tmp/lint/node_modules/.bin/browserslist, /tmp/lint/node_modules/.bin/cdk, /tmp/lint/node_modules/.bin/esparse, /tmp/lint/node_modules/.bin/esvalidate, /tmp/lint/node_modules/.bin/import-local-fixture, /tmp/lint/node_modules/.bin/jest, /tmp/lint/node_modules/.bin/js-yaml, ...

I check !

[nvuillam]

@ashokm I think i got it, PR on the way :)

[nvuillam]

@ashokm please could confirm it's ok with beta version ?

[ashokm]

Hi @nvuillam, I can confirm that the problem no longer exists in the latest beta version and the results are as expected (same as in v6).

Tested beta image:

 - Image Creation Date: 2023-09-20T17:50:24Z
 - Image Revision: 5bcd5083473a8d49bb315fbb45db89c8a6fe7827
 - Image Version: beta

[nvuillam]

@ashokm perfect, thanks for the feedback :) I've been really busy and for the moment I'm the only one handling releases, so there has been none for more than a month, but as I got COVID I have time to manage a new minor release soon, including this fix :)

[ashokm]

Hope you have a speedy recovery 🚀

[burner1024]

Looks like it's broken again, or was never fixed.

❌ Linted [COPYPASTE] files with [jscpd]: Found 145 error(s) - (62.29s)
- Using [jscpd v3.5.10] https://megalinter.io/7.6.0/descriptors/copypaste_jscpd
- MegaLinter key: [COPYPASTE_JSCPD]
- Rules config: [.jscpd.json]
--Error detail:
Clone found (javascript):
 - .vscode-test/vscode-1.43.2/VSCode-linux-x64/resources/app/out/vs/workbench/contrib/webview/electron-browser/pre/electron-index.js [1:1 - 6:187] (5 lines, 327 tokens)
   .vscode-test/vscode-1.44.2/VSCode-linux-x64/resources/app/out/vs/workbench/contrib/webview/electron-browser/pre/electron-index.js [1:1 - 6:187]

$ grep .vscode .gitignore 
.vscode-test/

[nvuillam]

@burner1024 if you want to ignore files with jscpd, you need to update .jscpd.json as this is not a linter that takes a list of files as input :)

https://megalinter.io/beta/configuration/#cli-lint-mode

[burner1024]

I just ran the command and expected IGNORE_GITIGNORED_FILES to be applied to all linters, as the doc suggests. Is there a list of linters that are ignoring IGNORE_GITIGNORED_FILES setting?

[nvuillam]

We could indeed update the documentation to precise that it only applies to linters that works with list of files mode (and not run without list of files on the repository root)

Most of linters handle by themselves to ignore files in .gitignore, but some of them like JSCPD do not, as in the example of our internal jscpd documentation -> https://megalinter.io/latest/descriptors/copypaste_jscpd/

Please feel free to post an issue on jscpd repo to request such enhancement :)

[burner1024]

I see, thank you for explanation.