search

terraform-linters / tflint-plugin-sdk

Experimental: TFLint plugin SDK for building custom rules
Mozilla Public License 2.0
17 stars 14 forks source link

how to retrieve or iterate all files #119

Closed universam1 closed 3 years ago

universam1 commented 3 years ago

I need to low-level iterate all files for style checking, but I can't see the list being accessible. I read the intention of File(string) is such a use case, also there is an unexported list under runner.data.Files available, but I'm struggling to grasp their access.

Use case is to determine if a heredoc syntax has been used.

Thank you for help @wata727

universam1 commented 3 years ago

Feels like a runner.WalkFiles() or simpliy a runner.Files() accessor is missing

wata727 commented 3 years ago

Currently, there is no API provided to access the low-level HCL body, but I think it is possible to provide such an API.

universam1 commented 3 years ago

thank you for the reply! Would you accept an PR or what would be the plan @wata727 ?

wata727 commented 3 years ago

Sure! This issue will be resolved someday, but if you need features sooner, please open a PR and I'll be happy to review it.

jonathansp commented 3 years ago

I've just opened this PR https://github.com/terraform-linters/tflint/pull/1132 which I imagine is the first step to have Files() available to the sdk. If the implementation is correct and approved, then I will add this to the client side.

wata727 commented 3 years ago

Nice! It's a good idea to merge the sdk changes first and then update the TFLint side PR.

nlamirault commented 3 years ago

Hi, I try this code :

func (rule *TerraformPortefaixStandardStructureRule) checkDirectories(runner tflint.Runner) error {
    files, _ := runner.Files()
    log.Printf("[INFO] Files: %d", len(files))
    allowedFiles := map[string]bool{"providers.tf": true, "main.tf": true}

    for name := range files {
        _, filename := path.Split(name)
        log.Printf("[INFO] OK: %s %s", name, filename)
        if _, exists := allowedFiles[filename]; !exists {
            message := fmt.Sprintf("File %s is not allowed here.", filename)

            return runner.EmitIssue(rule, message, hcl.Range{Start: hcl.InitialPos})
        }
    }
}

But i've got no files. Any idea ?

❯ ls -aRl
drwxr-xr-x nicolas users  4.0 KB Mon Jun 28 16:10:26 2021  .
drwxr-xr-x nicolas users  4.0 KB Wed Mar  3 10:57:09 2021  ..
.rw-r--r-- nicolas users    6 B  Mon Jun 28 16:10:26 2021  .terraform-version
.rwxr-xr-x nicolas users  2.5 KB Fri Mar 19 17:49:02 2021  .terraform.lock.hcl
drwxr-xr-x nicolas users  4.0 KB Fri Mar 19 17:49:02 2021  backend-vars
.rw-r--r-- nicolas users  691 B  Fri Mar 19 17:49:02 2021  backend.tf
.rw-r--r-- nicolas users  682 B  Fri Mar 19 17:49:02 2021  data.tf
.rw-r--r-- nicolas users 1013 B  Fri Mar 19 17:49:02 2021  dns.tf
.rwxr-xr-x nicolas users  772 B  Thu Jun 17 09:14:10 2021  main.tf
.rw-r--r-- nicolas users  615 B  Fri Mar 19 17:49:02 2021  outputs.tf
.rw-r--r-- nicolas users  685 B  Fri Mar 19 17:49:02 2021  provider.tf
drwxr-xr-x nicolas users  4.0 KB Fri Mar 19 17:49:02 2021  tfvars
.rw-r--r-- nicolas users  1.4 KB Fri Mar 19 17:49:02 2021  variables.tf

./backend-vars:
drwxr-xr-x nicolas users 4.0 KB Fri Mar 19 17:49:02 2021  .
drwxr-xr-x nicolas users 4.0 KB Mon Jun 28 16:10:26 2021  ..
.rw-r--r-- nicolas users 649 B  Fri Mar 19 17:49:02 2021  prod.tfvars

./tfvars:
drwxr-xr-x nicolas users  4.0 KB Fri Mar 19 17:49:02 2021  .
drwxr-xr-x nicolas users  4.0 KB Mon Jun 28 16:10:26 2021  ..
.rw-r--r-- nicolas users 1017 B  Fri Mar 19 17:49:02 2021  prod.tfvars

❯ tflint -c ../../../../.tflint.hcl --loglevel=trace
09:53:14 config.go:96: [INFO] Load config: ../../../../.tflint.hcl
09:53:14 config.go:311: [DEBUG] Config loaded
09:53:14 config.go:312: [DEBUG]   Module: false
09:53:14 config.go:313: [DEBUG]   Force: false
09:53:14 config.go:314: [DEBUG]   IgnoreModules: map[string]bool{}
09:53:14 config.go:315: [DEBUG]   Varfiles: []string{}
09:53:14 config.go:316: [DEBUG]   Variables: []string{}
09:53:14 config.go:317: [DEBUG]   DisabledByDefault: false
09:53:14 config.go:318: [DEBUG]   Rules: map[string]*tflint.RuleConfig{"terraform_portefaix_standard_structure":(*tflint.RuleConfig)(0xc0005866c0)}
09:53:14 config.go:319: [DEBUG]   Plugins: map[string]*tflint.PluginConfig{"portefaix":(*tflint.PluginConfig)(0xc0005866f0)}
09:53:14 option.go:49: [DEBUG] CLI Options
09:53:14 option.go:50: [DEBUG]   Module: false
09:53:14 option.go:51: [DEBUG]   Force: false
09:53:14 option.go:52: [DEBUG]   IgnoreModules: map[string]bool{}
09:53:14 option.go:53: [DEBUG]   EnableRules: []string(nil)
09:53:14 option.go:54: [DEBUG]   DisableRules: []string(nil)
09:53:14 option.go:55: [DEBUG]   Only: []string(nil)
09:53:14 option.go:56: [DEBUG]   EnablePlugins: []string(nil)
09:53:14 option.go:57: [DEBUG]   Varfiles: []string{}
09:53:14 option.go:58: [DEBUG]   Variables: []string{}
09:53:14 loader.go:57: [INFO] Initialize new loader
09:53:14 loader.go:82: [INFO] Load configurations under .
09:53:14 loader.go:90: [INFO] Module inspection is disabled. Building a root module without children...
09:53:14 loader.go:170: [INFO] Load values files
09:53:14 runner.go:50: [INFO] Initialize new runner for root
09:53:14 terraform.go:158: [INFO] TF_VAR_* environment variable found: key=TF_VAR_master_authorized_networks
09:53:14 discovery.go:68: [INFO] Plugin `portefaix` found
2021-07-02T09:53:14.891+0200 [DEBUG] plugin: starting plugin: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix args=[/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix]
2021-07-02T09:53:14.891+0200 [DEBUG] plugin: plugin started: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix pid=1378465
2021-07-02T09:53:14.891+0200 [DEBUG] plugin: waiting for RPC address: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix
2021-07-02T09:53:14.896+0200 [DEBUG] plugin.tflint-ruleset-portefaix: plugin address: address=/tmp/nicolas/plugin540353126 network=unix timestamp=2021-07-02T09:53:14.896+0200
2021-07-02T09:53:14.896+0200 [DEBUG] plugin: using plugin: version=8
09:53:14 provider.go:62: [INFO] Prepare rules
09:53:14 provider.go:90: [INFO]   3 default rules enabled
09:53:14 terraform_deprecated_interpolation.go:48: [TRACE] Check `terraform_deprecated_interpolation` rule for `root` runner
09:53:14 terraform_module_pinned_source.go:72: [TRACE] Check `terraform_module_pinned_source` rule for `root` runner
09:53:14 terraform_module_pinned_source.go:81: [DEBUG] Walk `dns.source` attribute
09:53:14 terraform_workspace_remote.go:48: [TRACE] Check `terraform_workspace_remote` rule for `root` runner
2021-07-02T09:53:14.900+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 09:53:14 [ERR] Check `terraform_portefaix_standard_structure` rule for runner
2021-07-02T09:53:14.900+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 09:53:14 [INFO] Files: 0
2021-07-02T09:53:14.900+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 09:53:14 [ERR] plugin: plugin server: accept unix /tmp/nicolas/plugin540353126: use of closed network connection
2021-07-02T09:53:14.901+0200 [DEBUG] plugin: plugin process exited: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix pid=1378465
2021-07-02T09:53:14.901+0200 [DEBUG] plugin: plugin exited
jonathansp commented 3 years ago

Hi @nlamirault could you please provide which version of tflint and tflint-plugin-sdk you currently have installed? according to the docs, tflint does not recursively find files, but it should list files from the root/current working dir:

Does TFLint check modules recursively?

No. TFLint always checks only the current root module (no recursive check). However, you can check calling child modules based on module arguments by enabling Module Inspection. This allows you to check that you are not passing illegal values to the module.
Note that if you want to recursively inspect local modules, you need to run them in each directory. This is a limitation that occurs because Terraform always works for one directory. TFLint tries to emulate Terraform's semantics, so cannot perform recursive inspection.
nlamirault commented 3 years ago 
$ cat go.mod
module github.com/nlamirault/tflint-ruleset-portefaix

go 1.16

require (
        github.com/agext/levenshtein v1.2.2 // indirect
        github.com/apparentlymart/go-textseg/v12 v12.0.0 // indirect
        github.com/fatih/color v1.10.0 // indirect
        github.com/golang/protobuf v1.4.3 // indirect
        github.com/google/go-cmp v0.5.6 // indirect
        github.com/hashicorp/go-plugin v1.4.1 // indirect
        github.com/hashicorp/go-version v1.3.0 // indirect
        github.com/hashicorp/hcl/v2 v2.10.0 // indirect
        github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d // indirect
        github.com/kr/text v0.2.0 // indirect
        github.com/kylelemons/godebug v1.1.0 // indirect
        github.com/mitchellh/go-testing-interface v1.14.1 // indirect
        github.com/mitchellh/go-wordwrap v1.0.0 // indirect
        github.com/sergi/go-diff v1.2.0 // indirect
        github.com/stretchr/testify v1.7.0 // indirect
        github.com/terraform-linters/tflint-plugin-sdk v0.8.3-0.20210614125323-8364139f3745 // indirect
        github.com/zclconf/go-cty v1.8.3 // indirect
        golang.org/x/net v0.0.0-20210326060303-6b1517762897 // indirect
        golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
        google.golang.org/appengine v1.6.6 // indirect
        google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d // indirect
        google.golang.org/grpc v1.32.0 // indirect
        google.golang.org/protobuf v1.25.0 // indirect
        gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
)

and : 

❯ tflint --version
TFLint version 0.28.1
+ ruleset.azurerm (0.8.2)
+ ruleset.google (0.5.1)
+ ruleset.aws (0.2.1)
jonathansp commented 3 years ago

missing a return nil at the end could be the cause? after the for loop

nlamirault commented 3 years ago

@jonathansp i've got it :) juste copy paste missing on github.

jonathansp commented 3 years ago
  1. I've just created a tf-files dir:
ls -la
total 0
drwxr-xr-x   5 jonathan 1052847317  160 Jul  2 09:30 .
drwxr-xr-x  13 jonathan 1052847317  416 Jul  2 09:30 ..
-rw-r--r--   1 jonathan 1052847317    0 Jul  2 09:30 dns.tf
-rw-r--r--   1 jonathan 1052847317    0 Jul  2 09:30 main.tf
-rw-r--r--   1 jonathan 1052847317    0 Jul  2 09:30 providers.tf

  1. created a new plugin called test, using your code.

  2. $ make install

  3. $ tflint --enable-plugin=test tf-files --only=test_rule

  4. the result was

1 issue(s) found:

Error: File dns.tf is not allowed here. (test_rule)

  on  line 1:
   (source code not available)

do you think it could be caused for any other reason, like relative paths or anything else I can do to reproduce the problem?

nlamirault commented 3 years ago

I push my code here : https://github.com/nlamirault/tflint-ruleset-portefaix Could you try it ? on the same directory.

jonathansp commented 3 years ago 
$ git clone https://github.com/nlamirault/tflint-ruleset-portefaix.git
Cloning into 'tflint-ruleset-portefaix'...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 38 (delta 9), reused 25 (delta 7), pack-reused 0
Unpacking objects: 100% (38/38), done.

$ cd tflint-ruleset-portefaix

$ make install
go build
mkdir -p ~/.tflint.d/plugins
mv ./tflint-ruleset-portefaix ~/.tflint.d/plugins

$ tflint-ruleset-portefaix git:(main)  tflint --enable-plugin=portefaix ../tf-files --only=terraform_portefaix_standard_structure
1 issue(s) found:

Error: File dns.tf is not allowed here. (terraform_portefaix_standard_structure)

  on  line 1:
   (source code not available)

Reference: https://github.com/nlamirault/tflint-ruleset-portefaix/blob/master/README.md
$ cat go.mod

module github.com/nlamirault/tflint-ruleset-portefaix

go 1.16

require (
    github.com/agext/levenshtein v1.2.3 // indirect
    github.com/fatih/color v1.12.0 // indirect
    github.com/google/go-cmp v0.5.6 // indirect
    github.com/hashicorp/go-hclog v0.16.1 // indirect
    github.com/hashicorp/go-plugin v1.4.2 // indirect
    github.com/hashicorp/go-version v1.3.0 // indirect
    github.com/hashicorp/hcl/v2 v2.10.0
    github.com/hashicorp/yamux v0.0.0-20210316155119-a95892c5f864 // indirect
    github.com/mattn/go-isatty v0.0.13 // indirect
    github.com/mitchellh/go-testing-interface v1.14.1 // indirect
    github.com/mitchellh/go-wordwrap v1.0.1 // indirect
    github.com/oklog/run v1.1.0 // indirect
    github.com/terraform-linters/tflint-plugin-sdk v0.8.3-0.20210614125323-8364139f3745
    github.com/vmihailenco/tagparser v0.1.2 // indirect
    github.com/zclconf/go-cty v1.8.4 // indirect
    golang.org/x/net v0.0.0-20210614182718-04defd469f4e // indirect
    golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c // indirect
    google.golang.org/appengine v1.6.7 // indirect
    google.golang.org/genproto v0.0.0-20210701191553-46259e63a0a9 // indirect
    google.golang.org/grpc v1.39.0 // indirect
)
$ tflint --version
TFLint version 0.29.0

I noticed you're using tflint 0.28, which doesn't expose Files() method yet. Update to 0.29 will solve this issue.

nlamirault commented 3 years ago 
$ ls -alFrt
drwxr-xr-x nicolas users 4.0 KB Fri Jul  2 14:23:58 2021  ../
.rw-r--r-- nicolas users   0 B  Fri Jul  2 14:24:03 2021  foo.tf
.rw-r--r-- nicolas users   0 B  Fri Jul  2 14:24:07 2021  doe.tf
drwxr-xr-x nicolas users 4.0 KB Fri Jul  2 14:24:07 2021  ./

$ tflint --version                                                                                         
TFLint version 0.29.1

$ tflint . --loglevel=debug --enable-plugin=portefaix --enable-rule=terraform_portefaix_standard_structure 
14:27:11 config.go:105: [INFO] Load config: .tflint.hcl
14:27:11 config.go:117: [INFO] Default config file is not found. Ignored
14:27:11 config.go:126: [INFO] Load fallback config: /home/nicolas/.tflint.hcl
14:27:11 config.go:134: [INFO] Fallback config file is not found. Ignored
14:27:11 config.go:136: [INFO] Use default config
14:27:11 option.go:50: [DEBUG] CLI Options
14:27:11 option.go:51: [DEBUG]   Module: false
14:27:11 option.go:52: [DEBUG]   Force: false
14:27:11 option.go:53: [DEBUG]   IgnoreModules: map[string]bool{}
14:27:11 option.go:54: [DEBUG]   EnableRules: []string{"terraform_portefaix_standard_structure"}
14:27:11 option.go:55: [DEBUG]   DisableRules: []string(nil)
14:27:11 option.go:56: [DEBUG]   Only: []string(nil)
14:27:11 option.go:57: [DEBUG]   EnablePlugins: []string{"portefaix"}
14:27:11 option.go:58: [DEBUG]   Varfiles: []string{}
14:27:11 option.go:59: [DEBUG]   Variables: []string{}
14:27:11 loader.go:57: [INFO] Initialize new loader
14:27:11 loader.go:82: [INFO] Load configurations under .
14:27:11 loader.go:90: [INFO] Module inspection is disabled. Building a root module without children...
14:27:11 loader.go:170: [INFO] Load values files
14:27:11 runner.go:50: [INFO] Initialize new runner for root
14:27:11 terraform.go:158: [INFO] TF_VAR_* environment variable found: key=TF_VAR_master_authorized_networks
14:27:11 discovery.go:90: [DEBUG] Find plugin path: /home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix
14:27:11 discovery.go:54: [INFO] Plugin `portefaix` found
2021-07-02T14:27:11.255+0200 [DEBUG] plugin: starting plugin: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix args=[/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix]
2021-07-02T14:27:11.255+0200 [DEBUG] plugin: plugin started: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix pid=1693657
2021-07-02T14:27:11.255+0200 [DEBUG] plugin: waiting for RPC address: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix
2021-07-02T14:27:11.262+0200 [DEBUG] plugin.tflint-ruleset-portefaix: plugin address: network=unix address=/tmp/nicolas/plugin443987081 timestamp=2021-07-02T14:27:11.262+0200
2021-07-02T14:27:11.262+0200 [DEBUG] plugin: using plugin: version=8
14:27:11 provider.go:62: [INFO] Prepare rules
14:27:11 provider.go:90: [INFO]   3 default rules enabled
2021-07-02T14:27:11.266+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 14:27:11 [DEBUG] Check `terraform_portefaix_standard_structure` rule for runner
2021-07-02T14:27:11.266+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 14:27:11 [DEBUG] Files: 0
2021-07-02T14:27:11.266+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 14:27:11 [ERR] plugin: plugin server: accept unix /tmp/nicolas/plugin443987081: use of closed network connection
2021-07-02T14:27:11.268+0200 [DEBUG] plugin: plugin process exited: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix pid=1693657
2021-07-02T14:27:11.268+0200 [DEBUG] plugin: plugin exited

I have no idea why it does not work :(

jonathansp commented 3 years ago

I think I may have been using tflint from master branch, which would explain that. @wata727 can you please confirm if this feature was release already or will be?

nlamirault commented 3 years ago

It works with master branch ! Cool.

2021-07-02T15:05:42.924+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 15:05:42 [ERR] Check `terraform_portefaix_standard_structure` rule for runner
2021-07-02T15:05:42.933+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 15:05:42 [INFO] Files: 7
2021-07-02T15:05:42.933+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 15:05:42 [INFO] OK: iac/gcp/dns/terraform/backend.tf backend.tf
1 issue(s) found:

Error: File backend.tf is not allowed here. (terraform_portefaix_standard_structure)

  on  line 1:
   (source code not available)

Reference: https://github.com/nlamirault/tflint-ruleset-portefaix/blob/master/README.md
wata727 commented 3 years ago

TFLint v0.30 / Plugin SDK v0.9 has been released.