Feature request: Docker image to run against multiple directories

[wyardley]

Since tflint doesn't support a recursive mode, for use in CI, it would useful if there were a variant of the Docker container that contains some way to check multiple files (e.g., a shell, such that something like find . -type f -name "*.tf" -exec dirname {} \; | uniq | xargs -n1 tflint can be run). I can see why it might not be ideal to build that into the default Dockefile, but some other lint tools (e.g., shellcheck) provide both options to support situations like this.

Or even an alternate entrypoint that supports checking a recusive filesystem hierarchy natively.

[bendrucker]

but some other lint tools (e.g., shellcheck) provide both options to support situations like this

If you have specific examples you can link to that I would be helpful. There's discussion in https://github.com/terraform-linters/tflint/pull/841. I'm generally skeptical that TFLint should be solving this. Even more so of maintaining it in shell/a Dockerfile.

[wyardley]

https://hub.docker.com/r/koalaman/shellcheck-alpine comes to mind https://github.com/koalaman/shellcheck#installing has a bit more context about how it's used.

I guess the question is, if I have a whole bunch of different states / directories to check, how would I handle that in CircleCI with the current Docker image, aside from creating a single workflow per directory / state?

Currently, we just bootstrap in a generic Terraform image by downloading the zip; had briefly moved to using the Docker image until I realized today that it doesn't actually operate recursively, the way most linters do.

A feature to accept multiple directories as arguments or to auto-discover directories would also resolve this issue, so if that WIP is going to go somewhere, that would definitely resolve the issue from my standpoint.

[bendrucker]

I guess the question is, if I have a whole bunch of different states / directories to check, how would I handle that in CircleCI with the current Docker image, aside from creating a single workflow per directory / state?

Use the Docker image and just run your own custom command inside.

In your shellcheck example I don't see any recursive processing, I just see shell globbing:

shellcheck myscripts/*.sh

Terraform is not comparable though, because files don't matter, directories do.

I don't #841 landing either b/c you need to run terraform init before invoking tflint --module. Which speaks to the idea that module finding should be handled by a separate application/script rather than in an individual analysis tool.

For now the best answer is BYO shell script but if there were a solid binary application that could handle this we could certainly recommend it as well as build it into the Dockerfile. I've considered working on one but have never gotten around to it.

[wyardley]

Use the Docker image and just run your own custom command inside.

Got it. I didn't see that the entrypoint could be overridden. Assuming the busybox utilities work Ok for my purposes, that should be good enough.

For installing plugins, is the recommendation to use wget directly?

[bendrucker]

Yep, I believe something like Circle is always going to try to invoke a shell wrapper around anything with run. For plugins, yes, use wget or curl. There's no discovery mechanism or standard there yet.

[wyardley]

Yep, I believe something like Circle is always going to try to invoke a shell wrapper around anything with run. For plugins, yes, use wget or curl. There's no discovery mechanism or standard there yet.

right, the issue is that there's no curl (which is what I was using before) in the container, and wget is the busybox one. That said, I was able to get it to work

  tflint:
    docker:
      - image: wata727/tflint:0.20.2
        entrypoint: /bin/sh ## not sure if it's needed, given your comment above
    steps:
      - checkout
      - run:
          name: Setup GCP plugin
          command: |
            cd /tmp
            wget -q https://github.com/terraform-linters/tflint-ruleset-google/releases/download/v0.1.0/tflint-ruleset-google_linux_amd64.zip
            mkdir -p ~/.tflint.d/plugins
            unzip tflint-ruleset-google_linux_amd64.zip
            cp tflint-ruleset-google ~/.tflint.d/plugins/
      - run:
          name: Check Terraform configurations with tflint
          command: |
            RETCODE=0
            for DIR in $(find . -type f -name "*.tf" -exec dirname {} \; | sort -u) ; do echo $DIR ; tflint $DIR || RETCODE=$((RETCODE+$?)) ; done
            if [[ $RETCODE -ge 1 ]]; then exit $RETCODE ; fi

[bendrucker]

Nice!

So is there something we can change about the images to make this a bit easier, short of providing the recursive script? Your example points at some of the potential desire for options like bailing on the first error or reporting all results, which pushes me away from shell.

Adding curl so you don't have to use busybox? Other utilities?

[wyardley]

Having curl, if feasible, would definitely be handy. And maybe even the install_plugin script from the project itself for extra convenience?

I guess if one wanted to validate modules as well, it would be hard without terraform itself, but without linux brew or similar, wouldn't be super trivial to allow the user to quickly get the needed terraform version (for that reason alone, I may keep on validating in a terraform docker image vs tflint).

Agree that keeping it as small and lightweight as possible is probably a good idea.

[bendrucker]

Ok, adding curl is no problem.

For installing Terraform, we could include https://github.com/tfutils/tfenv as an alternative to building an image with TFLint + Terraform.

Plugin installation I'm going to leave off for now because there is no spec for publishing/downloading plugins that script can rely on, even if the plugins in the terraform-linters org currently all follow the same pattern. A script, even just in the readme, ends up creating support issues, maintenance costs, etc. In the long run TFLint will move in the same direction as Terraform with providers, with a generic core, 100% pluggable rulesets, and officially supported schemes for downloading those plugins. It's top of mind but there's still a lot of work to get there.

[wyardley]

Yeah, tfenv would definitely be handy if that’s do-able

[bendrucker]

Hi, see the discussion in #928. We're not going to do this for now, but might consider providing a dedicated image with TFLint + accompanying tools for cloud use, potentially including:

• An installer like tfenv
• A utility for invoking a command in each module in a Terraform monorepo

I do understand the need, we have some similar functionality implemented via shell on CircleCI at @takescoop. I'll spend some time brainstorming ideas/specs for a Terraform monorepo CLI soon, and eventually that might make its way into a new offering that bundles TFLint with complimentary tooling.

[wyardley]

@bendrucker awesome. Thanks!