cloudposse / github-action-matrix-extended

GitHub Action that when used together with reusable workflows makes it easier to workaround the limit of 256 jobs in a matrix
https://cloudposse.com/accelerate
Apache License 2.0
5 stars 0 forks source link

JQ fails when affected stacks is empty #9

Open brett-au opened 5 months ago

brett-au commented 5 months ago

Describe the Bug

When updating our atmos repo, if we do not touch any component, catalog or stack files JQ fails to correctly parse the output.

jq: error: syntax error, unexpected ';' (Unix shell quoting issues?) at <top-level>, line 1:
def name: ;          
jq: 1 compile error
Error: The process '/usr/bin/jq' failed with exit code 3

When I run atmos locally the output is [] as expected

➜  affected=$(jq -c '.' affected-stacks.json)

➜  echo $affected
[]
➜  echo $affected | jq
[]

we are running cloudposse/github-action-atmos-affected-stacks@v3 which calls this action

Expected Behavior

cloudposse/github-action-atmos-affected-stacks@v3 runs successfully and returns an empty set so if: ${{ needs.context.outputs.has-affected-stacks == 'true' }} returns false correct and doesn't run more jobs based on the matrix.

Steps to Reproduce

name: "terraform plan matrix"

permissions:
  id-token: write
  contents: read
  pull-requests: write

on:
  pull_request:
    types: [opened, synchronize, reopened]

env:
  ATMOS_LOGS_LEVEL: Off

jobs:
  context:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - id: affected
      uses: cloudposse/github-action-atmos-affected-stacks@v3
      with:
        nested-matrices-count: 1

    outputs:
      affected: ${{ steps.affected.outputs.affected }}
      has-affected-stacks: ${{ steps.affected.outputs.has-affected-stacks }}
      matrix: ${{ steps.affected.outputs.matrix }}

Screenshots

image

image

Environment (please complete the following information):

Github actions runner ubuntu-latest

Additional Context

Add any other context about the problem here.

goruha commented 5 months ago

@brett-au I can not reproduce the bug.

Can you provide full logs pls

goruha commented 5 months ago

@brett-au have you config atmos.yaml` for gitops correctly?

https://github.com/cloudposse/github-action-atmos-affected-stacks/tree/main?tab=readme-ov-file#config

nitrocode commented 5 months ago

I'm getting the same error

integrations:
  github:
    gitops:
      terraform-version: 1.8.1
      infracost-enabled: false
      artifact-storage:
        region: us-east-1
        bucket: snip-tfstate
        # table: null
        role: arn:aws:iam::snip:role/snip
      role:
        plan: arn:aws:iam::snip:role/snip
        apply: arn:aws:iam::snip:role/snip
      matrix:
        sort-by: .stack_slug
        group-by: .stack_slug | split("-") | [.[0], .[2]] | join("-")

Logs

  if [[ "false" == "true" ]]; then
    atmos describe affected --file affected-stacks.json --verbose=true --repo-path "$GITHUB_WORKSPACE/main-branch" --include-spacelift-admin-stacks=true
  else
    atmos describe affected --file affected-stacks.json --verbose=true --repo-path "$GITHUB_WORKSPACE/main-branch"
  fi
  affected=$(jq -c '.' affected-stacks.json)
  printf "%s" "affected=$affected" >> $GITHUB_OUTPUT
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    ATMOS_LOGS_LEVEL: Off
    ATMOS_CLI_CONFIG_PATH: 
Found ENV var ATMOS_LOGS_LEVEL=Off
Found ENV var ATMOS_LOGS_LEVEL=Off
Found ENV var ATMOS_LOGS_LEVEL=Off
Current working repo HEAD: 59b525a959f242bec2da1dadb76ac22b90b2757a HEAD
Remote repo HEAD: 6cfa3819832e0d1f20b7e5af802dc8e90579cc51 refs/heads/main
Getting current working repo commit object...
Got current working repo commit object
Getting current working repo commit tree...
Got current working repo commit tree
Getting remote repo commit object...
Got remote repo commit object
Getting remote repo commit tree...
Got remote repo commit tree
Finding diff between the current working branch and remote target branch ...
Found diff between the current working branch and remote target branch
Changed files:
components/terraform/s3/main.md
stacks/catalog/terraform/s3/bucket-one.yaml
stacks/orgs/org/_defaults.yaml
Affected components and stacks:
Run cloudposse/github-action-matrix-extended@v0
Run cloudposse/github-action-jq@v0
/usr/bin/jq -c def name: ;
def group: ;
def nestedMatricesLevelsAre(v): 1 == v; 
def chunk: .key / 256 | floor;
def groupName: [(.value | group), (chunk | if . == 0 then empty else . end)] | join(" ");
def groupRange: [first, last] | map(.value | name) | join(" - ");
def toGHAMatrix: {"include": . } ;
def oneLevelMatrix: . ;
def twoLevelMatrix: 
    group_by(group) |
    map( to_entries | group_by(chunk) | 
        map({
            "name": first | groupName,
            "items": map(.value) | toGHAMatrix | tostring
        })
    ) | flatten;
def threeLevelMatrix:
    group_by(group) |
    map({
      "name": first | group,
      "items": ( to_entries | group_by(chunk) | 
        map({
          "name": groupRange,
          "items": map(.value) | toGHAMatrix | tostring
        }) | toGHAMatrix | tostring
      )
    });
def groupingStrategy:
    if nestedMatricesLevelsAre(3) then 
        threeLevelMatrix
    elif nestedMatricesLevelsAre(2)  then 
        twoLevelMatrix 
    else 
        oneLevelMatrix 
    end; 
def parseInput:
    if type == "array" then . else .include end;
parseInput | sort_by(name) | groupingStrategy | toGHAMatrix affected-stacks.json
jq: error: syntax error, unexpected ';' (Unix shell quoting issues?) at <top-level>, line 1:
def name: ;          
jq: 1 compile error
Error: The process '/usr/bin/jq' failed with exit code 3
nitrocode commented 5 months ago

I see the issue.

⨠ atmos describe config -f json | jq .
parse error: Invalid numeric literal at line 1, column 6

which fails because of these 2 outputs shown

⨠ atmos describe config -f json | head -2
Found ENV var ATMOS_BASE_PATH=/localhost/git/work/atmos
Found ENV var ATMOS_BASE_PATH=/localhost/git/work/atmos

This is because

# atmos.yaml
logs:
  level: Trace

I also tried setting ATMOS_LOGS_LEVEL and get the same error

⨠ export ATMOS_LOGS_LEVEL=Off
⨠ atmos describe config -f json | jq .
parse error: Invalid numeric literal at line 1, column 6

Options

  1. Set log level Off in atmos.yaml and set ATMOS_LOGS_LEVEL=Trace in geodesic
  2. or fix atmos to allow using an environment variable to turn off the Found ENV in the output
  3. or update to strip out Found ENV from the output

I think I’ll just do (1) for now but I’d prefer (2) and (3) seems like a temp fix