search

CoveredCA / common-devops

Apache License 2.0
0 stars 0 forks source link

update scan dependencies action so that when a new scan deps issue is created it has a status of "Todo" rather than no status #7

Closed tallguyjenks closed 2 weeks ago

tallguyjenks commented 1 month ago

file to update: https://github.com/CoveredCA/common-devops/blob/main/.github/workflows/scan-dependencies.yml

code to modify for intended results:

            gh -R "${{ github.repository }}" issue create \
            -t "scan-dependencies update ${{ env.service_name }} ${{ env.service_version }}" \
            -a "$configuration_scandependencies_assignees" \
            -b "${{ env.ISSUE_MESSAGE }}" \
            -l "$scandependencies_issue_label" \
            -p "${{ env.deployment_project }}"
stevenquitugua commented 1 month ago

Research:

6 | deployment_project: Deployment
7 | badge_repository: badges
8 | custom_log_platform: Splunk
9 | scandependencies_issue_label: dependencies  --need to change this to "TODO"
tallguyjenks commented 1 month ago

No, this is incorrect that is the issue label, this work is regarding the issue STATUS which i was wrong with the command to modify, this will require a bit more involved work to update the issue status because you'll also need to find the issue ID to reference it first

in fact you may need to make a new sub action module for this functionality to be referenced in the scan dependencies job using the code from this action as templated pieces to achieve the end results

looks like you're gonna have some fun with the github graph API

https://github.com/CoveredCA/common-devops/blob/main/packages/update-deployment-issue/action.yml

stevenquitugua commented 1 month ago
stevenquitugua commented 1 month ago

Research

tallguyjenks commented 1 month ago

we should probably have a call about this, theres a lot of code to get lost in, in that file, and there are only a few of those steps that you'll need to update just the status field

tallguyjenks commented 4 weeks ago

please book a meeting with me to go over this issue when you come back to it, there are some pieces you can pull from the other action but there will be a little code to mess with

stevenquitugua commented 3 weeks ago
tallguyjenks commented 3 weeks ago

duplicate folder and rename to update-scan-dependencies-issue please

remove references to deployment-issues it should be scan-dependency-issues nothing about deployment in this new action

tallguyjenks commented 3 weeks ago

the case of the status is exactly what is displayed which i just checked is Todo one word, proper case with the only capital being the T

stevenquitugua commented 3 weeks ago

@tallguyjenks Here is the first draft of the code for common-devops/packages/update-deployment-issue/action.yml

Code:

name: Update Deployment Issue
description: Deployment issues need custom project fields filled out
inputs:
  organization:
    required: false
    default: "CoveredCA"
    description: Name of the organization the project is in
  project-number:
    required: false
    default: 5
    description: The number of the project from the project web url
  application-name:
    required: true
    default: "cca-salesforce-sapi"
    description: The name of the application the deployment issue was created for
  environment:
    required: true
    default: "dev"
    description: The enmviropnment the deployment issue is calling for
  version:
    required: true
    default: "1.0.10"
    description: The version of the application to be deployed to the environment
  token:
    required: true
    description: The token for the mulebot so that github api actions can be performed

runs:
  using: "composite"
  steps:
    - name: Split Version Fields
      id: version-fields
      shell: bash
      run: |
        echo ""
        echo "*************************"
        echo "* Split Version Numbers *"
        echo "*************************"
        echo ""
        VERSION_MAJOR=$(echo "${{ inputs.version }}" | cut -d "." -f 1)
        VERSION_MINOR=$(echo "${{ inputs.version }}" | cut -d "." -f 2)
        VERSION_PATCH=$(echo "${{ inputs.version }}" | cut -d "." -f 3)
        echo "VERSION_MAJOR == $VERSION_MAJOR"
        echo "VERSION_MINOR == $VERSION_MINOR"
        echo "VERSION_PATCH == $VERSION_PATCH"
        echo "VERSION_MAJOR=$VERSION_MAJOR" >> $GITHUB_OUTPUT
        echo "VERSION_MINOR=$VERSION_MINOR" >> $GITHUB_OUTPUT
        echo "VERSION_PATCH=$VERSION_PATCH" >> $GITHUB_OUTPUT

    - name: Get project ID
      id: project-id
      shell: bash
      env:
        GITHUB_TOKEN: ${{ inputs.token }}
      run: |
        echo ""
        echo "******************"
        echo "* Get Project ID *"
        echo "******************"
        echo ""
        PROJECT_ID=$(gh api graphql -f query='{
          organization(login: "${{ inputs.organization }}") {
            login
            projectsV2(first: 100) {
              edges {
                node {
                  id
                  title
                }
              }
            }
          }
        }' | jq -r '.data.organization.projectsV2.edges[] | select(.node.title | test("Deployment")) | .node.id')

        echo "PROJECT_ID=$PROJECT_ID" >> $GITHUB_OUTPUT

    - name: Get issue ID
      id: issue-id
      shell: bash
      env:
        GITHUB_TOKEN: ${{ inputs.token }}
      run: |
        echo ""
        echo "****************"
        echo "* Get Issue ID *"
        echo "****************"
        echo ""

        # Initial cursor is empty
        CURSOR=""

        # Loop until all pages are fetched
        while : ; do
            # Execute GraphQL query
            RESPONSE=$(gh api graphql -f query='
            query($org: String!, $number: Int!, $cursor: String) {
                organization(login: $org) {
                    projectV2(number: $number) {
                        items(first: 100, after: $cursor) {
                            pageInfo {
                                hasNextPage
                                endCursor
                            }
                            nodes {
                                id
                                content {
                                    ... on Issue {
                                        id
                                        number
                                        title
                                    }
                                }
                            }
                        }
                    }
                }
            }' -f org="${{ inputs.organization }}" -F number=${{ inputs.project-number }} -f cursor="$CURSOR")

            # Parse items and do something with them, e.g., append to a file
            echo "$RESPONSE" | jq '.data.organization.projectV2.items.nodes[]' >> scan-dependency-issues.json

            # Check if there's a next page; if not, break the loop
            HAS_NEXT_PAGE=$(echo "$RESPONSE" | jq '.data.organization.projectV2.items.pageInfo.hasNextPage')
            if [ "$HAS_NEXT_PAGE" != "true" ]; then
                break
            fi

            # Update cursor to fetch next page
            CURSOR=$(echo "$RESPONSE" | jq -r '.data.organization.projectV2.items.pageInfo.endCursor')
        done

        echo "All items fetched successfully."

        echo "Parsing output data file"

        ISSUE_ID=$(jq -r '. | select(.content.title | test("scan-dependencies")) | select(.content.title | test("${{ inputs.application-name }}")) | select(.content.title | test("${{ inputs.version }}")) | .id' scan-dependency-issues.json)

        echo "ISSUE_ID=$ISSUE_ID" >> $GITHUB_OUTPUT

    - name: Get field IDs
      id: field-id
      shell: bash
      env:
        GITHUB_TOKEN: ${{ inputs.token }}
      run: |
        echo ""
        echo "******************"
        echo "* Get Field ID's *"
        echo "******************"
        echo ""

        FIELD_DATA=$(gh api graphql -f query='
                    query($org: String!, $number: Int!) {
                      organization(login: $org){
                        projectV2(number: $number) {
                          id
                          fields(first:20) {
                            nodes {
                              ... on ProjectV2Field {
                                id
                                name
                              }
                            ... on ProjectV2IterationField {
                            id
                            name
                            configuration {
                                iterations {
                                startDate
                                id
                                }
                            }
                            }
                              ... on ProjectV2SingleSelectField {
                                id
                                name
                                options {
                                  id
                                  name
                                }
                              }
                            }
                          }
                        }
                      }
                    }' -f org="${{ inputs.organization }}" -F number=${{ inputs.project-number }} | jq '.data.organization.projectV2.fields.nodes[] | select(.name | IN("Status"))')

        FIELD_ID_STATUS=$(echo "$FIELD_DATA" | jq -r '. | select(.name | test("Status")).id')
        FIELD_ID_STATUS_TODO_ID=$(echo "$FIELD_DATA" | jq -r '. | select(.name | test("Status")).options[] | select(.name=="Todo").id')

        echo "FIELD_ID_STATUS=$FIELD_ID_STATUS" >> $GITHUB_OUTPUT
        echo "FIELD_ID_STATUS_TODO_ID=$FIELD_ID_STATUS_TODO_ID" >> $GITHUB_OUTPUT

    - name: Update Issue Status Field
      shell: bash
      env:
        GITHUB_TOKEN: ${{ inputs.token }}
      run: |
        echo ""
        echo "*******************************"
        echo "* Updating Issue Status Field *"
        echo "*******************************"
        echo ""
        gh api graphql -f query='
            mutation {
                updateProjectV2ItemFieldValue(
                    input: {
                        projectId: "${{steps.project-id.outputs.PROJECT_ID}}"
                        itemId: "${{ steps.issue-id.outputs.ISSUE_ID }}"
                        fieldId: "${{ steps.field-id.outputs.FIELD_ID_STATUS }}"
                        value: {
                            singleSelectOptionId: "${{ steps.field-id.outputs.FIELD_ID_STATUS_TODO_ID }}"
                        }
                    }
                )
                {
                    projectV2Item {
                        id
                    }
                }
            }'
tallguyjenks commented 3 weeks ago
  1. common-devops/packages/update-deployment-issue/action.yml should be common-devops/packages/update-scan-dependencies-issue/action.yml
  2. remove the environment input
  3. remove the following step from the code - name: Split Version Fields

after that, looking great! 👍🏻 to implement and test. there IS actually a dependency to update in the parent pom so you can test by simply running the scan-dependencies job on the parent pom repository

stevenquitugua commented 3 weeks ago

✔ Code pushed

tallguyjenks commented 3 weeks ago

you dont need to bump versions, you can just run the scan dependencies job because its failing right now and making issues.

after each time you create a scan-dependencies issue by running the job, close it afterwards for subsequent tests

also, you will need to add the update issue step to the scan dependencies job in .github/workflows/scan-dependencies.yml file so the logic is actually used

stevenquitugua commented 3 weeks ago

you dont need to bump versions, you can just run the scan dependencies job because its failing right now and making issues. after each time you create a scan-dependencies issue by running the job, close it afterwards for subsequent tests also, you will need to add the update issue step to the scan dependencies job in .github/workflows/scan-dependencies.yml file so the logic is actually used

I put the following code into scan-dependencies.yml, but it failed, the only thing I removed from it was the environment:

name: Update deployment issue Todo fields
  uses: CoveredCA/common-devops/packages/update-scan-dependencies-issue@main
  with:
    token: ${{ env.github_automationbot_token }}
    application-name: ${{ env.service_name }}
    version: ${{ env.deployment_version }}
tallguyjenks commented 3 weeks ago

https://github.com/CoveredCA/common-devops/blob/03994861e1e3c75e23aedd070ccd3d601dfe4973/packages/update-scan-dependencies-issue/action.yml#L114C212-L114C235

you didnt update the name of the json file at the end of the line so its looking for something that doenst exists at present

stevenquitugua commented 3 weeks ago

https://github.com/CoveredCA/common-devops/blob/03994861e1e3c75e23aedd070ccd3d601dfe4973/packages/update-scan-dependencies-issue/action.yml#L114C212-L114C235

you didnt update the name of the json file at the end of the line so its looking for something that doenst exists at present

I renamed from deployment_issues.json to scan-dependency-issues.json, change it back to deployment_issues.json?

The video of our meeting at 00:36s said to change the name of the .json though

tallguyjenks commented 3 weeks ago

what the heck was i seeing, you're good, that is correct, i could have sworn i saw it was named differently 🤷🏻‍♂️ let me look at the actions log

stevenquitugua commented 3 weeks ago

what the heck was i seeing, you're good, that is correct, i could have sworn i saw it was named differently 🤷🏻‍♂️ let me look at the actions log

I could try to put the following code before the Create Badge step:

      - name: Update deployment issue Todo fields
        uses: CoveredCA/common-devops/packages/update-scan-dependencies-issue@main
        with:
          token: ${{ env.github_automationbot_token }}
          application-name: ${{ env.service_name }}
          version: ${{ env.deployment_version }}

Currently it is after create badge so maybe that is the issue? In scan-dependencies.yml.

tallguyjenks commented 3 weeks ago

figured it out!

in https://github.com/CoveredCA/common-devops/blob/main/.github/workflows/scan-dependencies.yml

      - name: Create badge
        uses: CoveredCA/common-devops/packages/badge@main
        if: always()
        with:
          label: ${{ env.BADGE_LABEL}}
          status: ${{ env.dependencies_badge_status }}
          color: ${{ env.dependencies_badge_color }}
          file: ${{ env.service_name }}-dependencies.svg
          folder: ${{ env.service_name }}

      - name: Update deployment issue Todo fields
        uses: CoveredCA/common-devops/packages/update-scan-dependencies-issue@main
        with:
          token: ${{ env.github_automationbot_token }}
          application-name: ${{ env.service_name }}
          version: ${{ env.deployment_version }}

at the bottom, see that line if: always() when scan dependencies fails the rest of the steps are considered unneeded because the job failed already, but always() means it "always" runs which is desired for the badge because we're only running this code when scan dependencies fails, however we dont want it to "always" run because sometimes scan dependencies succeeds and there's nothing to update and no issue to update, so we need to only run your new code when scan dependencies fails

so add this to your step:

if: failure() && steps.integrationTests.outcome == 'failure'
      - name: Update deployment issue Todo fields
        uses: CoveredCA/common-devops/packages/update-scan-dependencies-issue@main
        if: failure() && steps.scan-dependencies.outcome == 'failure'
        with:
          token: ${{ env.github_automationbot_token }}
          application-name: ${{ env.service_name }}
          version: ${{ env.deployment_version }}

and add this to the scan dependencies step:

before:

      - name: Scan maven dependencies
        shell: bash
        run: |

after:

      - name: Scan maven dependencies
        id: scan-dependencies
        shell: bash
        run: |
tallguyjenks commented 3 weeks ago

whoops, forgot to update the copy and paste

the addition should be if: failure() && steps.scan-dependencies.outcome == 'failure' if it wasnt clear

stevenquitugua commented 3 weeks ago

Failed: The failure appears to be related to line 114 in update-scan-dependencies-issue /action.yml

Research:

Replacing: 

ISSUE_ID=$(jq -r '. | select(.content.title | test("scan-dependencies")) | select(.content.title | test("${{ inputs.application-name }}")) | select(.content.title | test("${{ inputs.version }}")) | .id' scan-dependency-issues.json)

WITH:

ISSUE_ID=$(jq -r ". | select(.content.title | test(\"scan-dependencies\")) | select(.content.title | test(\"${{ inputs.application-name }}\")) | select(.content.title | test(\"${{ inputs.version }}\")) | .id" scan-dependency-issues.json)

Re-run Scan dependencsies:

stevenquitugua commented 3 weeks ago

Log from when I had printed out scan-dependencies-issues.json: C:\Users\squitugua\Downloads\logs_27614388454

tallguyjenks commented 2 weeks ago

think i found the current problem:

image

our jq query is grabbing .id which in this example is PVTI_lADOAfwVjc4AV0R8zgSOcqo or an ID of similar length, like the top level id field in the json object:

{
  "id": "PVTI_lADOAfwVjc4AV0R8zgSOcqo",
  "content": {
    "id": "I_kwDOKXNuts6URCq0",
    "number": 48,
    "title": "scan-dependencies update common-parent-pom 1.1.3"
  }
}

but thats the wrong ID format for issues, i think the beginning of the ID's are also indicators of that, PVII being indicative of "Project V2" and the I_ being indicative of "Issue"

so in this case we need to modify the jq query from looking at just .id to .content.id

so line 114 from

        ISSUE_ID=$(jq -r '. | select(.content.title | test("scan-dependencies")) | select(.content.title | test("${{ inputs.application-name }}")) | select(.content.title | test("${{ inputs.version }}")) | .id' scan-dependency-issues.json)

to

        ISSUE_ID=$(jq -r '. | select(.content.title | test("scan-dependencies")) | select(.content.title | test("${{ inputs.application-name }}")) | select(.content.title | test("${{ inputs.version }}")) | .content.id' scan-dependency-issues.json)
stevenquitugua commented 2 weeks ago

✔Changed line 114 as suggested, and committed

❌Re-ran scan-dependencies and now have this error: Error: Invalid format 'I_kwDOKXNuts6Jxw0o'

tallguyjenks commented 2 weeks ago

wheres the action execution log? i dont see the latest run in the action history?

stevenquitugua commented 2 weeks ago

wheres the action execution log? i dont see the latest run in the action history?

Link to log: https://github.com/CoveredCA/common-parent-pom/actions/runs/10564276080/job/29626444769

stevenquitugua commented 2 weeks ago

Dependencies to be updated if not already:

[INFO] The following dependencies in Dependency Management have newer versions:
[INFO]   com.microsoft.sqlserver:mssql-jdbc ........ 12.8.0.jre8 -> 12.8.1.jre8
[INFO]   com.microsoft.sqlserver:mssql-jdbc_auth ..... 12.8.0.x64 -> 12.8.1.x64
[INFO]   net.snowflake:snowflake-jdbc ........................ 3.18.0 -> 3.19.0
[INFO]   org.mule.connectors:mule-db-connector ............. 1.14.11 -> 1.14.12
[INFO]   org.mule.connectors:mule-sftp-connector ............... 2.2.1 -> 2.2.2
tallguyjenks commented 2 weeks ago

oh i get it you re-ran an older action ✔️

tallguyjenks commented 2 weeks ago

the issue version didnt flow, i think that caused the latest failure:

image image

tallguyjenks commented 2 weeks ago

i think i found the fix

in .github/workflows/scan-dependencies.yml

change the last line from:

          version: ${{ env.deployment_version }}

to

          version: ${{ env.service_version }}
tallguyjenks commented 2 weeks ago

make that change then re-run job

stevenquitugua commented 2 weeks ago

Re-running scan-dep:

tallguyjenks commented 2 weeks ago

okay, lets revert the .content.id back to .id on line 114

stevenquitugua commented 2 weeks ago

okay, lets revert the .content.id back to .id on line 114

Re-ran: Error: Invalid format 'PVTI_lADOAfwVjc4AV0R8zgPQt0A'

tallguyjenks commented 2 weeks ago

im gonna run some local tests, hold please

tallguyjenks commented 2 weeks ago

okay, i added some more echo messages to validate that things were working as intended, the version is now flowing correctly, (we need to run new instance of scan-dependencies each time not just rerunning the older job) and it succeeded everywhere until it got to the next step, the Get field IDs step

image

image

at least i think this is the case

tallguyjenks commented 2 weeks ago

okay i manually set everything up in vscode and ran it all and successfully updated the field, but its weird why its not working in the action, by all accounts it should be

tallguyjenks commented 2 weeks ago

well it worked and is fixed. only change was updating the version of the parent pom and triggering the scan deps job. so at present, this issue is now resolved.

im forseeing potentially there may be issue if the the scan deps job runs on the schedule and open an issue on its own. but we'll get there when we get there. call this work completed and fill out the project fields ✔️

tallguyjenks commented 2 weeks ago

image