search

jbergstroem / hadolint-gh-action

A hadolint linter for github actions that provides code annotations (and more)
MIT License
12 stars 5 forks source link

Shell expansion **/Dockerfile not finding file recursively #81

Closed mloskot closed 2 years ago

mloskot commented 2 years ago

For a repo with collection of Dockerfile-s structured like this:

/a/x/Dockerfile
/a/y/Dockerfile
/a/z/Dockerfile

the following workflow with shell expansion

jobs:
  hadolint:
    name: Dockerfiles
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: jbergstroem/hadolint-gh-action@v1
      with:
        dockerfile: "**/Dockerfile"

fails with

hadolint: **/Dockerfile: openBinaryFile: does not exist (No such file or directory)

image

Is the recursive look-up for multiple Dockerfile-s even supported? I failed to find any note on that in the docs.

jbergstroem commented 2 years ago

Thanks for the elaborate report! The glob expansion should be working through the running shell; I'd call this a bug. Lets see what we can do!

jbergstroem commented 2 years ago

@mloskot I just opened a draft PR to address this (needs some shellcheck fixes) - perhaps you can try it and let me know how it goes? You should be able to replace v1 with fix/expand-glob-prior-hadolint when using the action.

mloskot commented 2 years ago

@jbergstroem Thank you for light-speed quick fix. AFAICT, there is no difference:

jobs:
  docker:
    name: Dockerfiles
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: jbergstroem/hadolint-gh-action@fix/expand-glob-prior-hadolint
      with:
        dockerfile: "**/Dockerfile"

image

jbergstroem commented 2 years ago

@jbergstroem Thank you for light-speed quick fix. AFAICT, there is no difference

Thanks for helping out to debug! Could you pass an environment variable called DEBUG? It should tell you what bash commands are executed and ultimately if the glob is expanded:

# Adapted from the added test in test/e2e.sh:
$ DEBUG=1 output_format=tty dockerfile=test/fixtures/**/Dockerfile-glob* ./hadolint.sh
# some noise
+ local OUTPUT=
++ eval hadolint --no-fail --no-color '' -f json 'test/fixtures/**/Dockerfile-glob*'
+++ hadolint --no-fail --no-color -f json test/fixtures/glob-path/Dockerfile-glob-1 test/fixtures/glob-path/Dockerfile-glob-2
# more noise
mloskot commented 2 years ago

@jbergstroem Sure, here it is the output

Run jbergstroem/hadolint-gh-action@fix/expand-glob-prior-hadolint
  with:
    dockerfile: **/Dockerfile
    error_level: 0
    annotate: true
  env:
    DEBUG: YES

Run VERSION=$(</home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/HADOLINT_VERSION)
  VERSION=$(</home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/HADOLINT_VERSION)
  mkdir /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/bin
  wget -q -O /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/bin/hadolint \
    "[https://github.com/hadolint/hadolint/releases/download/v${VERSION}/hadolint-Linux-x86_64](https://github.com/hadolint/hadolint/releases/download/v$%7BVERSION%7D/hadolint-Linux-x86_64)"
  chmod +x /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/bin/hadolint
  echo "/home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/bin" >> $GITHUB_PATH
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    DEBUG: YES

Run /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/hadolint.sh
  /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/hadolint.sh
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    DEBUG: YES
    dockerfile: **/Dockerfile
    config_file: 
    error_level: 0
    annotate: true
    output_format: 
+ set -euo pipefail nullglob
+++ dirname /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/hadolint.sh
++ cd /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint
++ pwd
+ DIR=/home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint
+ . /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/lib/hadolint.sh
++ HADOLINT_PATH=hadolint
+ . /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/lib/main.sh
++ DOCKERFILE='**/Dockerfile'
++ CONFIG_FILE=
++ ERRORLEVEL=0
++ ANNOTATE=true
++ OUTPUT_FORMAT=
++ HADOLINT_PATH=hadolint
+ . /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/lib/jq.sh
+ . /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/lib/validate.sh
+ run
+ for executable in "${HADOLINT_PATH}" jq
+ command -v hadolint
+ for executable in "${HADOLINT_PATH}" jq
+ command -v jq
+ output_hadolint_version
+ local OUTPUT=
++ eval hadolint --version
++ cut -d ' ' -f 4
+++ hadolint --version
+ OUTPUT=2.10.0
+ echo '::set-output name=hadolint_version::2.10.0'
+ validate_error_level 0
+ for error_level in -1 0 1 2
+ [[ -1 == \0 ]]
+ for error_level in -1 0 1 2
+ [[ 0 == \0 ]]
+ return 0
+ validate_annotate true
+ [[ true == \t\r\u\e ]]
+ return 0
+ [[ -z '' ]]
+ local CONFIG=
+ [[ -z '' ]]
+ local DOCKERFILES=
+ DOCKERFILES=("${DOCKERFILE}")
+ [[ -n '' ]]
+ local OUTPUT=
++ eval hadolint --no-fail --no-color '' -f json '**/Dockerfile'
+++ hadolint --no-fail --no-color -f json '**/Dockerfile'
hadolint: **/Dockerfile: openBinaryFile: does not exist (No such file or directory)
+ OUTPUT=
Error: Process completed with exit code 1.
jbergstroem commented 2 years ago

Ok, so it doesn't expand it. I think I have enough info to reproduce in a test repo, but the interesting bit here is why the environment acts differently.

jbergstroem commented 2 years ago

@mloskot I think expanding by merging the array should be more successful. Can you try latest version of the branch?

mloskot commented 2 years ago

@jbergstroem

so it doesn't expand it.

AFAIK,set -x is not sticky across calls to sub-scripts/shells.

I think the DEBUG env should also trigger export SHELLOPTS.

However, I've tried to run the latest:


env:
  DEBUG: "YES"

jobs:
  docker:
    name: Dockerfiles
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: jbergstroem/hadolint-gh-action@fix/expand-glob-prior-hadolint
      with:
        dockerfile: "**/Dockerfile"
Run jbergstroem/hadolint-gh-action@fix/expand-glob-prior-hadolint
  with:
    dockerfile: **/Dockerfile
    error_level: 0
    annotate: true
  env:
    DEBUG: YES
Run VERSION=$(</home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/HADOLINT_VERSION)
  VERSION=$(</home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/HADOLINT_VERSION)
  mkdir /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/bin
  wget -q -O /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/bin/hadolint \
    "[https://github.com/hadolint/hadolint/releases/download/v${VERSION}/hadolint-Linux-x86_64](https://github.com/hadolint/hadolint/releases/download/v$%7BVERSION%7D/hadolint-Linux-x86_64)"
  chmod +x /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/bin/hadolint
  echo "/home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/bin" >> $GITHUB_PATH
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    DEBUG: YES
Run /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/hadolint.sh
  /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/hadolint.sh
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    DEBUG: YES
    dockerfile: **/Dockerfile
    config_file: 
    error_level: 0
    annotate: true
    output_format: 
+ set -euo pipefail nullglob
+++ dirname /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/hadolint.sh
++ cd /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint
++ pwd
+ DIR=/home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint
+ . /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/lib/hadolint.sh
++ HADOLINT_PATH=hadolint
+ . /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/lib/main.sh
++ DOCKERFILE='**/Dockerfile'
++ CONFIG_FILE=
++ ERRORLEVEL=0
++ ANNOTATE=true
++ OUTPUT_FORMAT=
++ HADOLINT_PATH=hadolint
+ . /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/lib/jq.sh
+ . /home/runner/work/_actions/jbergstroem/hadolint-gh-action/fix/expand-glob-prior-hadolint/lib/validate.sh
+ run
+ for executable in "${HADOLINT_PATH}" jq
+ command -v hadolint
+ for executable in "${HADOLINT_PATH}" jq
+ command -v jq
+ output_hadolint_version
+ local OUTPUT=
++ eval hadolint --version
++ cut -d ' ' -f 4
+++ hadolint --version
+ OUTPUT=2.10.0
+ echo '::set-output name=hadolint_version::2.10.0'
+ validate_error_level 0
+ for error_level in -1 0 1 2
+ [[ -1 == \0 ]]
+ for error_level in -1 0 1 2
+ [[ 0 == \0 ]]
+ return 0
+ validate_annotate true
+ [[ true == \t\r\u\e ]]
+ return 0
+ [[ -z '' ]]
+ local CONFIG=
+ [[ -z '' ]]
+ [[ -n '' ]]
+ local OUTPUT=
++ eval hadolint --no-fail --no-color '' -f json '**/Dockerfile'
+++ hadolint --no-fail --no-color -f json '**/Dockerfile'
hadolint: **/Dockerfile: openBinaryFile: does not exist (No such file or directory)
+ OUTPUT=
Error: Process completed with exit code 1
jbergstroem commented 2 years ago

Thanks for testing. I figured out that you can use use: ./ in github actions to refer to your own repo; so the idea is to build out some test cases for this as part of the PR - will take a look a bit later.

Edit:

AFAIK, set -x is not sticky across calls to sub-scripts/shells.

True, but I source the files into hadolint.sh (otherwise you wouldn't see that output).

jbergstroem commented 2 years ago

Got a test in my PR now for this: https://github.com/jbergstroem/hadolint-gh-action/actions/runs/3099210690/jobs/5018078307

Looks like single quoting variables probably is the culprit; will dig in.

jbergstroem commented 2 years ago

So, after a lot of debugging I figured it out.. the default running options in the ubuntu environment doesn't have globstar or nullglob enabled. Best part is that I learned a lot and were able to set up a e2e integration test for this.

jbergstroem commented 2 years ago

@mloskot thanks a lot for your help. Feel free to test @master and report back.

Given:

$ ls test/**/Dockerfile-glob-*
test/fixtures/glob-path/Dockerfile-glob-1  test/fixtures/glob-path/Dockerfile-glob-2

..this expands properly: https://github.com/jbergstroem/hadolint-gh-action/actions/runs/3100095777/jobs/5019985427#step:3:79

mloskot commented 2 years ago

@jbergstroem I can see the action collects all my Dockerfiles 

hadolint --no-fail --no-color -f json MyApp/ServerCore/Dockerfile MyServer/Alpine/Dockerfile MyClient/ServerCore/Dockerfile

Thank you a lot for your prompt support!

jbergstroem commented 2 years ago

I will fix a few small things and have a release by the end of this week.