antonbabenko / pre-commit-terraform

pre-commit git hooks to take care of Terraform configurations πŸ‡ΊπŸ‡¦
MIT License
3.15k stars 533 forks source link

[Other pre-commit hooks in pre-commit-terraform Docker image] github action error during installation #359

Closed diegolagospagopa closed 2 years ago

diegolagospagopa commented 2 years ago

Hi People,

Configurations

I have this configurations

repos:
  ## general
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.2.0
    hooks:
      # Common errors
      - id: end-of-file-fixer
        exclude_types: [sql]
      - id: trailing-whitespace
        args: [--markdown-linebreak-ext=md]
        exclude_types: [sql]
      - id: check-yaml
      - id: check-executables-have-shebangs
      # Cross platform
      - id: check-case-conflict
      - id: mixed-line-ending
        args: [--fix=lf]
        exclude_types: [sql]
      # Security
      - id: detect-aws-credentials
        args: ['--allow-missing-credentials']
      - id: detect-private-key
  ## terraform
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.64.1
    hooks:
      - id: terraform_fmt
      - id: terraform_docs
        args:
          - --hook-config=--path-to-file=README.md        # Valid UNIX path. I.e. ../TFDOC.md or docs/README.md etc.
          - --hook-config=--add-to-existing-file=true     # Boolean. true or false
          - --hook-config=--create-file-if-not-exist=true # Boolean. true or false
          - --args=--hide providers
      - id: terraform_tfsec
      - id: terraform_validate

Pipeline

name: Static Analysis

on:
  push:

jobs:
  static_analysis:
    runs-on: ubuntu-20.04

    steps:
      - uses: actions/checkout@v2

      - name: init_src_policy
        env:
          FOLDERS: policy
        run: |
          TAG=$(cat .terraform-version)
          for f in $FOLDERS; do
            cd src/$f
            sed -i -e 's/  backend "azurerm" {}//g' 99_main.tf # use local backend
            docker run -v $(pwd):/tmp -w /tmp hashicorp/terraform:$TAG init
          done

      - name: run_test
        run: |
          TAG=v1.64.1
          docker run -v $(pwd):/lint -w /lint ghcr.io/antonbabenko/pre-commit-terraform:$TAG run -a

Problem

Running this action, i find this error

Run TAG=v1.64.1
Unable to find image 'ghcr.io/antonbabenko/pre-commit-terraform:v1.64.1' locally
v1.64.1: Pulling from antonbabenko/pre-commit-terraform
[5](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:5)9bf1c3509f3: Already exists
07a400e93df3: Pulling fs layer
d8c[6](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:6)81bc2183: Pulling fs layer
c[7](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:7)f1ca1171d9: Pulling fs layer
dc9256fcea2e: Pulling fs layer
[8](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:8)2b2235ac423: Pulling fs layer
31ced[9](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:9)5c34e7: Pulling fs layer
5a3e01799d8c: Pulling fs layer
123b07747a77: Pulling fs layer
dc9256fcea2e: Waiting
82b2235ac423: Waiting
31ced95c34e7: Waiting
5a3e01799d8c: Waiting
123b07747a77: Waiting
843dab449416: Pulling fs layer
843dab449416: Waiting
c7f1ca1171d9: Verifying Checksum
c7f1ca1171d9: Download complete
07a400e93df3: Verifying Checksum
07a400e93df3: Download complete
d8c681bc2183: Verifying Checksum
d8c681bc2183: Download complete
dc9256fcea2e: Verifying Checksum
dc9256fcea2e: Download complete
82b2235ac423: Verifying Checksum
82b2235ac423: Download complete
07a400e93df3: Pull complete
123b07747a77: Verifying Checksum
123b07747a77: Download complete
5a3e01799d8c: Verifying Checksum
5a3e01799d8c: Download complete
31ced95c34e7: Verifying Checksum
31ced95c34e7: Download complete
843dab449416: Verifying Checksum
843dab449416: Download complete
d8c681bc2183: Pull complete
c7f1ca1171d9: Pull complete
dc9256fcea2e: Pull complete
82b2235ac423: Pull complete
31ced95c34e7: Pull complete
5a3e01799d8c: Pull complete
123b07747a77: Pull complete
843dab449416: Pull complete
Digest: sha256:f1e4797d02a89ab274e984ff26154d799c1209a642cc6ab362973790b2b6182a
Status: Downloaded newer image for ghcr.io/antonbabenko/pre-commit-terraform:v1.64.1
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/antonbabenko/pre-commit-terraform.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/root/.cache/pre-commit/repoibac5ccm/py_env-python3/bin/python', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout:
    Processing /root/.cache/pre-commit/repoibac5ccm
      Preparing metadata (setup.py): started
      Preparing metadata (setup.py): finished with status 'done'
    Collecting ruamel.yaml>=0.15
      Downloading ruamel.yaml-0.17.21-py3-none-any.whl ([10](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:10)9 kB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.5/109.5 KB 8.2 MB/s eta 0:00:00
    Collecting toml
      Downloading toml-0.10.2-py2.py3-none-any.whl ([16](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:16) kB)
    Collecting ruamel.yaml.clib>=0.2.6
      Downloading ruamel.yaml.clib-0.2.6.tar.gz (180 kB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 180.7/180.7 KB 23.0 MB/s eta 0:00:00
      Preparing metadata (setup.py): started
      Preparing metadata (setup.py): finished with status 'error'

stderr:
      error: subprocess-exited-with-error

      Γ— python setup.py egg_info did not run successfully.
      β”‚ exit code: 1
      ╰─> [3 lines of output]
          sys.argv ['/tmp/pip-install-htjtaeow/ruamel-yaml-clib_c5e3ea74f3e94e83b1944fd834e[17](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:17)1[72](https://github.com/pagopa/devops-azure-governance/runs/5991589876?check_suite_focus=true#step:4:72)/setup.py', 'egg_info', '--egg-base', '/tmp/pip-pip-egg-info-s8ovu009']
          test compiling /tmp/tmp_ruamel_at0lb4mm/test_ruamel_yaml.c -> test_ruamel_yaml compile error: /tmp/tmp_ruamel_at0lb4mm/test_ruamel_yaml.c
          Exception: command 'gcc' failed: No such file or directory
          [end of output]

      note: This error originates from a subprocess, and is likely not a problem with pip.
    error: metadata-generation-failed

    Γ— Encountered error while generating package metadata.
    ╰─> See above for output.

    note: This is an issue with the package mentioned above, not pip.
    hint: See above for details.

Check the log at /root/.cache/pre-commit/pre-commit.log
Error: Process completed with exit code 3.

see: pipeline log

This configuration works

repos:
  ## terraform
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.64.1
    hooks:
      - id: terraform_fmt
      - id: terraform_docs
        args:
          - --hook-config=--path-to-file=README.md        # Valid UNIX path. I.e. ../TFDOC.md or docs/README.md etc.
          - --hook-config=--add-to-existing-file=true     # Boolean. true or false
          - --hook-config=--create-file-if-not-exist=true # Boolean. true or false
          - --args=--hide providers
      - id: terraform_tfsec
      - id: terraform_validate

see https://github.com/pagopa/devops-azure-governance/runs/5991612215?check_suite_focus=true

Help

Please can you help me to understand what is the problem, because in local works as expected.

ps: in the logs there are some info about the fact that gcc is not found, but i make some test and gcc is in place.

Have a nice day

MaxymVlasov commented 2 years ago

Hi, try to force install

sudo apt update
sudo apt install build-essential

and if you got some additional errors like "have no package X" - try to install "X-dev" package.

Also, check that you use exactly python 3.

here are my common checks for infra, hope that will be helpfull

name: Common issues check

on: [pull_request]

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - run: |
        git fetch --no-tags --prune --depth=1 origin +refs/heads/*:refs/remotes/origin/*
    - name: Get changed files
      id: file_changes
      run: |
        export DIFF=$(git diff --name-only origin/${{ github.base_ref }} ${{ github.sha }})
        echo "Diff between ${{ github.base_ref }} and ${{ github.sha }}"
        echo "::set-output name=files::$( echo "$DIFF" | xargs echo )"

    - name: TFLint cache plugin dir
      uses: actions/cache@v2
      with:
        path: ~/.tflint.d/plugins
        key: ubuntu-latest-tflint-${{ hashFiles('.tflint.hcl') }}

    - name: Setup TFLint
      uses: terraform-linters/setup-tflint@v1.1.0

    - name: Init TFLint
      run: tflint --init

    - name: Setup Terraform docs
      env:
        # Can be 'latest' or valid version like '0.15.0'
        TERRAFORM_DOCS_VERSION: latest
      run: |
        ( \
            TERRAFORM_DOCS_RELEASES="https://api.github.com/repos/terraform-docs/terraform-docs/releases" && \
            [ "$TERRAFORM_DOCS_VERSION" = "latest" ] && curl -L "$(curl -s ${TERRAFORM_DOCS_RELEASES}/latest | grep -o -E -m 1 "https://.+?-linux-amd64.tar.gz")" > terraform-docs.tgz \
            || curl -L "$(curl -s ${TERRAFORM_DOCS_RELEASES} | grep -o -E "https://.+?v${TERRAFORM_DOCS_VERSION}-linux-amd64.tar.gz")" > terraform-docs.tgz \
        ) \
        && tar -xzf terraform-docs.tgz terraform-docs && rm terraform-docs.tgz \
        && chmod +x terraform-docs && sudo mv terraform-docs /usr/bin/

    - name: Install shfmt
      run: |
        curl -L "$(curl -s https://api.github.com/repos/mvdan/sh/releases/latest | grep -o -E -m 1 "https://.+?linux_amd64")" > shfmt \
        && chmod +x shfmt && sudo mv shfmt /usr/bin/

    - uses: hashicorp/setup-terraform@v1
      with:
        terraform_version: ~0.13.6

    - uses: actions/setup-python@v2
      with:
        python-version: '3.9'
    # Need to success pre-commit fix push
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0
        ref: ${{ github.event.pull_request.head.sha }}
    # Skip terraform_tflint which interferes to commit pre-commit auto-fixes
    - name: Execute pre-commit
      uses: pre-commit/action@v2.0.0
      env:
        SKIP: no-commit-to-branch,terraform_tflint
      with:
        token: ${{ secrets.GITHUB_TOKEN }}
        extra_args: --color=always --show-diff-on-failure --files ${{ steps.file_changes.outputs.files }}
    # Run only skipped checks
    - name: Execute pre-commit check that have no auto-fixes
      if: always()
      uses: pre-commit/action@v2.0.0
      with:
        extra_args: >
          --config .github/.pre-commit-hooks-without-autofixes.yaml
          --color=always --show-diff-on-failure --files ${{ steps.file_changes.outputs.files }}
diegolagospagopa commented 2 years ago

Thx so much i will test it

Jno21 commented 2 years ago

Hi,

I had the same issue. Like you guessed it is linked to https://github.com/pre-commit/pre-commit-hooks.

To solve this issue it is possible to just update alpine-sdk, this will add the dependencies that you need to run the container:

apk add --update alpine-sdk 

I am wondering more globally if we should include this in the Dockerfile of this project to be able to run those checks since they are pretty common, but that would also "create" a dependency with another repository / project which is complex and might not be what we want in this project.

antonbabenko commented 2 years ago

This issue has been resolved in version 1.70.0 :tada: