Low value automation actions import/removal identification

[devopsmatt]

There are a lot of calls to externally hosted GH workflows. This ticket is to examine the dependencies and create a list of functional actions and nice-to-have but not necessary for lean operation.

The goal is to reduce complexity and overall execution time of workflows run ad-hoc, on PRs etc.

DoD: listing of external dependencies that can be safely removed vs those that should be imported to our repo.

[emvaldes]

There are a lot of nested third parties involved in the overall architecture of GitHub Actions in this project. It will be a colossal endeavor to attempt importing all these but at least I am documenting this to have a basic understanding of how wide this issue is.

Actions-R-Us
ad-m
amannn
andymckay
avto-dev
aws-actions
bazelbuild
bewuethr
crazy-max
cycjimmy
dawidd6
docker
emibcn
gaurav-nelson
golfzaptw
gradle
grolston
guyarb
ianlewis
josiahsiegel
kreuzwerker
marocchino
nick-invision
ossf
paultyng
peter-evans
release-drafter
sigstore
slsa-framework
softprops
thehanimo
toolmantim

[emvaldes]

Nevertheless, we should focus in evaluating the following GitHub Actions and their value to the project and import them if it's decided to do so.

AzViz-action
action-connect-ovpn -> connect-ovpn (already imported)
checksum-validate-action
git-secrets
randomrepo
rapid-wsl
reliable-pull-request-action
remote-branch-action
runleaks
slack-boltjs-app
stackoverflow_in_pg
terraform-stats
terraform-templates
workflow-housekeeper

[emvaldes]

Profile: JosiahSiegel

Objective: Azure Visualizer aka 'AzViz' - PowerShell module to automatically generate Azure resource topology diagrams by just typing a PowerShell cmdlet and passing the name of one or more Azure Resource Group(s). Note: Cloud admins are not anymore doomed to manually document a cloud environment! The pain of inheriting an undocumented cloud landscape to support is gone. It is capable of:

• Finding Resources in a Azure Resource Group and identifying their dependencies.
• Plot nodes and edges to represent Azure Resources and their dependencies on a graph.
• Insert appropriate Azure Icons on basis of resource category/sub-category.
• Label each resource with information like Name, Category, Type etc.
• Generate visualization in formats like: .png and .svg
• Output image can be in 'light', 'dark' or 'neon' theme.
• Can target more than one resource group at once.
• Change direction in which resource groups are plotted, i.e, left-to-right or top-to-bottom.

Target: AzViz-action@v1.0.4 : 663e242 Latest: AzViz-action (0ae465075a8f0401db5256f1ea35193ee08ec687)

Invoked: Indirectly (nested)

./runleaks/.github/workflows/scan_public.yml:9:
default: '[\"josiahsiegel\/runleaks\",\"josiahsiegel\/AzViz-action\"]'

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16156 Warning: This is a clone to another repository PrateekKumarSingh/AzViz

[emvaldes]

Profile: JosiahSiegel

Objective: Determine if test string checksum valid or invalid.

1. Generate a checksum from either a string or shell command (use command substitution: $()).
2. Validate if checksum is identical to input (even across multiple jobs), using a key to link the validation attempt with the correct generated checksum. a. Validation is possible across jobs since the checksum is uploaded as a workflow artifact

Target: checksum-validate-action@v1.5 : ebdf8c1 Latest: checksum-validate-action (806ce2fa215d520071c6d4faf8d2588a65e23749)

Invoked: Directly

Workflows: ./.github/workflows/release_to_azure.yml:148:

Actions: ./.github/actions/deploy-backend/action.yml:336:

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16157 Note: Further development was made and not referenced/used in the project.

[emvaldes]

Profile: JosiahSiegel

Objective: Prevent committing AWS , AZURE and GCP sensitive credentials to a git repository.

Target: git-secrets@v1.3.0 : ad82d68 Latest: git-secrets -> (bff90684a2289cd3b0536a2cbeb37728e30680fd)

Invoked: Indirectly (nested)

./runleaks/Dockerfile:5:
RUN git clone https://github.com/JosiahSiegel/git-secrets.git

./runleaks/Dockerfile:6:
RUN make -C /git-secrets install

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16186 Warning: This is a fork to another repository msalemcode/git-secrets

[emvaldes]

Profile: JosiahSiegel

Objective: This action is useful for running actions against random repositories. e.g. leak scanner, etc.

Target: randomrepo@v1.1 : 443cdca

Invoked: Indirectly (nested)

./runleaks/.github/workflows/scan_public.yml:28:
uses: JosiahSiegel/randomrepo@v1.1

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16158 Warning: Generated fron JosiahSiegel/hello-world-docker-action

[emvaldes]

Profile: JosiahSiegel

Objective: Easily manage your WSL environment with a handful of generic commands. Applications are separated into modules. Remote repository is cloned into distro if specified (-r).

Target: rapid-wsl@latest : c9c5785 (latest)

Invoked: Indirectly (nested)

./remote-branch-action/.github/workflows/test-action.yml:26:
repository: josiahsiegel/rapid-wsl

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16177

[emvaldes]

Profile: JosiahSiegel

Objective: Creates a pull request on a GitHub repository using existing branches and then by executing the actions/checkout determines the active repo.

Target: reliable-pull-request-action@v1.2.0 : ae8d0c8

Invoked: Directly

Workflows: ./.github/workflows/prepare_deployment_branch.yaml:[31,40]:

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16159

[emvaldes]

Profile: JosiahSiegel

Objective: Creates a branch on a remote GitHub repository and then by executing the actions/checkout determines the active repo.

Target: remote-branch-action@v1.2.0 : ae8d0c8

Invoked: Directly

Workflows: ./github/workflows/prepare_deployment_branch.yaml:31:

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16161

[emvaldes]

Profile: JosiahSiegel

Objective: Leverages git-secrets to identify potential leaks in GitHub action run logs. Common Azure and Google Cloud patterns are available, thanks to fork msalemcode/git-secrets.

Target: runleaks@v1.3 : 4dd30d1

Invoked: Directly

Workflows: ./github/workflows/scan_action_logs.yml:[16,17,21,25,26]:

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16015

[emvaldes]

Profile: JosiahSiegel

Objective: A secure and simple BoltJS app for Slack ChatOps. The app includes basic GitHub push functionality to get you started. Helpful links

1. Bolt getting started guide
2. Bolt documentation
3. Slack app home

Target: slack-boltjs-app@v1.1.0 : 6cce33c Latest: slack-boltjs-app (a5f682a3381fc1b5c07dce1a6a99350175e15719)

Invoked: Directly

Workflows: ./.github/workflows/release_chatops_app.yml:[8,13,36]:

Actions: ./.github/actions/build-vars/action.yml:164:

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16160 Note: Further development was made and not referenced/used in the project.

[emvaldes]

Profile: JosiahSiegel

Objective: Import Stackoverflow database into a PostgreSQL database. The social network Stackoverflow (https://stackoverflow.com/) regularly publishes a dump of its database under a Creative Commons free license. We can find dump file here.

Target: stackoverflow_in_pg@latest : c4c3bbe (latest)

Invoked: Indirectly (nested)

./terraform-templates/azure/env/03/~locals.tf:98:
repos = {
    terraform-templates = {
        url = "https://github.com/JosiahSiegel/terraform-templates.git",
        mount_path = "/app/repo1" },
    so2pg = {
        url = "https://github.com/JosiahSiegel/stackoverflow_in_pg.git",
        mount_path = "/app/repo2"
    }
}

./terraform-templates/.scripts/data/psql_table_massive.sh:9:
python stackoverflow_in_pg/python_src/so2pg-posthistory.py PostHistory.xml \
> posthistory.sql

./terraform-templates/.scripts/data/psql_table_tiny.sh:9:
python stackoverflow_in_pg/python_src/so2pg-tags.py Tags.xml > tags.sql

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16204

[emvaldes]

Profile: JosiahSiegel

Objective: Output the following statistics for the Terraform environment:

1. Terraform version
2. Drift count a. "Drift" refers to changes made outside of Terraform and does not necessary match any resources listed for changes.
3. Resource drifts
4. Change count a. "Change" refers to change actions that Terraform plans to use to move from the prior state to a new state.
5. Change percent a. Percentage of changes to total resources.
6. Resource changes

Target: terraform-stats@v1.6 : 68b8cbe Latest: terraform-stats (c49292abedbdec5db97c73ab4f15613c9d359790)

Invoked: Directly

Workflows: ./.github/workflows/alert_terraform_changes.yml:[32,55]:

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16019 Note: Further development was made and not referenced/used in the project.

[emvaldes]

Profile: JosiahSiegel

Objective: This repository contains Terraform templates for deploying various resources in Azure. It is designed to help beginners get started with provisioning infrastructure using Terraform.

Target: terraform-templates@latest : a11c320 (latest)

Invoked: Indirectly (nested)

./rapid-wsl/defaults/demo.sh:1:
-d Ubuntu-22.04 \
-m demo \
-u demouser \
-r 'https://github.com/JosiahSiegel/terraform-templates.git'

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16187

[emvaldes]

Profile: JosiahSiegel

Objective: Retain a time period or quantity of workflow runs.

Target: workflow-housekeeper@v1.1.0 : 731cc20

Invoked: Directly

Workflows: ./.github/workflows/log_management.yml:[16,28]:

Tracking GitHub Issue: https://github.com/CDCgov/prime-reportstream/issues/16162

[emvaldes]

These are the GitHub Workflows and Actions we need to further review. Understanding these GitHub Workflows/Actions is critical for us so that we can identify the true nature/objectives and inter-dependencies while using these external GitHub Actions.

Clarifications: I am in favor to import them and leave them as they are until we can tackle these at a later time. I believe we should import both direct and indirect modules. In ideal conditions, I would recommend to simply fork them and create a functional branch to interconnect them all while maintaining an upstream link.

Workflows:

1. ./.github/workflows/alert_terraform_changes.yml:

   • josiahsiegel/terraform-stats@68b8cbe42c494333fbf6f8d90ac86da1fb69dcc2

2. ./.github/workflows/log_management.yml:

   • JosiahSiegel/workflow-housekeeper@731cc20bb613208b34efb6ac74aab4ba147abb50

3. ./.github/workflows/prepare_deployment_branch.yaml:

   • JosiahSiegel/remote-branch-action@dbe7a2138eb064fbfdb980abee918091a7501fbe
   • JosiahSiegel/reliable-pull-request-action@ae8d0c88126329ee363a35392793d0bc94cb82e7

4. ./.github/workflows/release_to_azure.yml:

   • JosiahSiegel/checksum-validate-action@ebdf8c12c00912d18de93c483b935d51582f9236

5. ./.github/workflows/release_chatops_app.yml:

   • focusconsulting/slack-boltjs-app -> JosiahSiegel/slack-boltjs-app

6. ./.github/workflows/scan_action_logs.yml:

   • josiahsiegel/runleaks@4dd30d107c03b6ade87978e10c94a77015e488f9

Actions:

• ./.github/actions/build-vars/action.yml
• ./.github/actions/deploy-backend/action.yml

[emvaldes]

There are a lot of multiple externally referenced GitHub Actions directly invoked within the project's pipelines. A very large and significant volume of dependencies that we do not control their compliance.

$ find . -type f \
   | xargs -I {} egrep -Hn -i "\ uses\:\ " {} \
   | egrep -v -i "josiahsiegel" \
   | egrep -v -i "\#\# DevSecOps \- Aquia \(Replace\) " \
   | awk -F':' '{print $1" -> "$3":"$4}' \
   | sed -e 's|\([[:blank:]]\)\{1,\}|\ |g' \
   | sort -u \
   | egrep -v "uses: \.\/\.github\/" \
   | egrep -v "uses: github\/" \
   | egrep -v "uses: actions\/" ;

Actions:

./.github/actions/build-auth/action.yml -> - uses: gradle/actions/setup-gradle@
./.github/actions/build-backend/action.yml -> - uses: gradle/actions/setup-gradle@
./.github/actions/build-backend/action.yml -> uses: EnricoMi/publish-unit-test-result-action/linux@
./.github/actions/build-backend/action.yml -> uses: jwalton/gh-docker-logs@
./.github/actions/build-submissions/action.yml -> - uses: gradle/actions/setup-gradle@
./.github/actions/build-vars/action.yml -> - uses: azure/login@
./.github/actions/build-vars/action.yml -> - uses: dorny/paths-filter@
./.github/actions/connect-ovpn/README.md -> uses: golfzaptw/action-connect-ovpn@master
./.github/actions/demo-env/action.yml -> uses: convictional/trigger-workflow-and-wait@
./.github/actions/notifications/action.yml -> uses: rtCamp/action-slack-notify@
./.github/actions/retry/action.yml -> - uses: nick-fields/retry@
./.github/actions/sonarcloud/action.yml -> uses: sonarsource/sonarcloud-github-action@
./.github/actions/vpn-azure/action.yml -> - uses: azure/login@

Workflows:

./.github/workflows/alert_PD_schedule_Slack.yml -> uses: antifree/json-to-variables@
./.github/workflows/alert_version_upgrade.yml -> uses: supriyaaddagada/json-to-variables@
./.github/workflows/build_hub.yml -> uses: peter-evans/create-or-update-comment@
./.github/workflows/deploy_terraform.yml -> uses: hashicorp/setup-terraform@
./.github/workflows/destroy_demo_environment.yml -> uses: hashicorp/setup-terraform@
./.github/workflows/frontend_chromatic_main.yml -> uses: chromaui/action@
./.github/workflows/frontend_ci.yml -> # uses: snyk/actions/node@
./.github/workflows/frontend_ci.yml -> uses: chromaui/action@
./.github/workflows/frontend_ci.yml -> uses: peter-evans/create-or-update-comment@
./.github/workflows/frontend_ci.yml -> uses: peter-evans/find-comment@
./.github/workflows/publish_docker.yaml -> - uses: dorny/paths-filter@
./.github/workflows/publish_docker.yaml -> uses: docker/login-action@
./.github/workflows/release_chatops_app.yml -> - uses: azure/login@
./.github/workflows/release_to_azure.yml -> uses: hashicorp/setup-terraform@
./.github/workflows/release_to_github.yml -> uses: dev-drprasad/delete-older-releases@
./.github/workflows/release_to_github.yml -> uses: mikepenz/release-changelog-builder-action@
./.github/workflows/release_to_github.yml -> uses: ncipollo/release-action@
./.github/workflows/restore_databases.yml -> - uses: azure/login@
./.github/workflows/snyk.yml -> - uses: gradle/actions/setup-gradle@
./.github/workflows/snyk.yml -> - uses: snyk/actions/setup@
./.github/workflows/sonarcloud.yml -> uses: gradle/actions/setup-gradle@
./.github/workflows/sonarcloud.yml -> uses: tj-actions/branch-names@
./.github/workflows/sonarcloud.yml -> uses: tj-actions/changed-files@
./.github/workflows/start_test_servers.yml -> - uses: azure/login@
./.github/workflows/stop_test_servers.yml -> - uses: azure/login@
./.github/workflows/validate_resources.yml -> - uses: azure/login@
./.github/workflows/validate_terraform.yml -> uses: bridgecrewio/checkov-action@
./.github/workflows/validate_terraform.yml -> uses: hashicorp/setup-terraform@

[emvaldes]

These are GitHub Actions hosted locally within the project and are directly invoked:

Actions:

./.github/actions/build-auth/action.yml -> uses: ./.github/actions/retry

./.github/actions/build-backend/action.yml -> uses: ./.github/actions/retry
./.github/actions/build-backend/action.yml -> uses: ./.github/actions/runner-ip
./.github/actions/build-backend/action.yml -> uses: ./.github/actions/vpn-azure

./.github/actions/build-frontend/action.yml -> uses: ./.github/actions/retry

./.github/actions/build-submissions/action.yml -> uses: ./.github/actions/retry
./.github/actions/db-restore/action.yml -> - uses: ./.github/actions/retry

./.github/actions/demo-env/action.yml -> uses: ./.github/actions/retry
./.github/actions/demo-env/action.yml -> uses: ./.github/actions/runner-ip

./.github/actions/deploy-backend/action.yml -> uses: ./.github/actions/retry

./.github/actions/vpn-azure/action.yml -> - uses: ./.github/actions/connect-ovpn

Workflows:

./.github/workflows/StaleItemsReport.yml -> uses: ./.github/actions/notifications

./.github/workflows/alert_MBUsers_Inactive.yml -> uses: ./.github/actions/notifications
./.github/workflows/alert_PD_schedule_Slack.yml -> uses: ./.github/actions/notifications
./.github/workflows/alert_cert_expire.yml -> uses: ./.github/actions/notifications
./.github/workflows/alert_cert_expire.yml -> uses: ./.github/actions/runner-ip
./.github/workflows/alert_cert_expire.yml -> uses: ./.github/actions/vpn-azure
./.github/workflows/alert_resource_costs.yml -> uses: ./.github/actions/az-cost
./.github/workflows/alert_resource_costs.yml -> uses: ./.github/actions/notifications
./.github/workflows/alert_resource_costs.yml -> uses: ./.github/actions/vpn-azure
./.github/workflows/alert_stale_branches.yaml -> uses: ./.github/actions/notifications
./.github/workflows/alert_terraform_changes.yml -> uses: ./.github/actions/notifications
./.github/workflows/alert_terraform_changes.yml -> uses: ./.github/actions/vpn-azure
./.github/workflows/alert_version_upgrade.yml -> uses: ./.github/actions/notifications

./.github/workflows/build_frontend.yaml -> uses: ./.github/actions/build-frontend
./.github/workflows/build_frontend.yaml -> uses: ./.github/actions/build-vars
./.github/workflows/build_hub.yml -> uses: ./.github/actions/build-backend
./.github/workflows/build_hub.yml -> uses: ./.github/actions/build-vars

./.github/workflows/cleanup_acr_images.yml -> uses: ./.github/actions/vpn-azure

./.github/workflows/deploy_terraform.yml -> uses: ./.github/actions/build-vars
./.github/workflows/deploy_terraform.yml -> uses: ./.github/actions/vpn-azure
./.github/workflows/deployment_rollback.yml -> uses: ./.github/actions/runner-ip
./.github/workflows/deployment_rollback.yml -> uses: ./.github/actions/vpn-azure

./.github/workflows/destroy_demo_environment.yml -> uses: ./.github/actions/demo-env
./.github/workflows/destroy_demo_environment.yml -> uses: ./.github/actions/vpn-azure

./.github/workflows/export_cost_data.yml -> uses: ./.github/actions/az-cost
./.github/workflows/export_cost_data.yml -> uses: ./.github/actions/db-query
./.github/workflows/export_cost_data.yml -> uses: ./.github/actions/runner-ip
./.github/workflows/export_cost_data.yml -> uses: ./.github/actions/vpn-azure

./.github/workflows/frontend_ci.yml -> uses: ./.github/actions/build-vars

./.github/workflows/release_chatops_app.yml -> uses: ./.github/actions/build-vars
./.github/workflows/release_to_azure.yml -> uses: ./.github/actions/build-backend
./.github/workflows/release_to_azure.yml -> uses: ./.github/actions/build-frontend
./.github/workflows/release_to_azure.yml -> uses: ./.github/actions/build-vars
./.github/workflows/release_to_azure.yml -> uses: ./.github/actions/demo-env
./.github/workflows/release_to_azure.yml -> uses: ./.github/actions/deploy-backend
./.github/workflows/release_to_azure.yml -> uses: ./.github/actions/deploy-frontend
./.github/workflows/release_to_azure.yml -> uses: ./.github/actions/vpn-azure
./.github/workflows/release_to_github.yml -> uses: ./.github/actions/build-backend
./.github/workflows/release_to_github.yml -> uses: ./.github/actions/build-frontend
./.github/workflows/release_to_github.yml -> uses: ./.github/actions/build-vars
./.github/workflows/release_trial_frontend.yml -> uses: ./.github/actions/build-frontend
./.github/workflows/release_trial_frontend.yml -> uses: ./.github/actions/build-vars
./.github/workflows/release_trial_frontend.yml -> uses: ./.github/actions/deploy-frontend
./.github/workflows/release_trial_frontend.yml -> uses: ./.github/actions/vpn-azure

./.github/workflows/restore_databases.yml -> uses: ./.github/actions/db-backup
./.github/workflows/restore_databases.yml -> uses: ./.github/actions/db-query
./.github/workflows/restore_databases.yml -> uses: ./.github/actions/db-restore
./.github/workflows/restore_databases.yml -> uses: ./.github/actions/mount-share
./.github/workflows/restore_databases.yml -> uses: ./.github/actions/retry

./.github/workflows/snyk.yml -> uses: ./.github/actions/build-vars

./.github/workflows/sonarcloud.yml -> uses: ./.github/actions/build-auth
./.github/workflows/sonarcloud.yml -> uses: ./.github/actions/build-submissions
./.github/workflows/sonarcloud.yml -> uses: ./.github/actions/retry
./.github/workflows/sonarcloud.yml -> uses: ./.github/actions/sonarcloud

./.github/workflows/start_test_servers.yml -> uses: ./.github/actions/Start-AzFunction
./.github/workflows/start_test_servers.yml -> uses: ./.github/actions/Start-PostgresDB

./.github/workflows/stop_test_servers.yml -> uses: ./.github/actions/Stop-AzFunction
./.github/workflows/stop_test_servers.yml -> uses: ./.github/actions/Stop-PostgresDB
./.github/workflows/stop_test_servers.yml -> uses: ./.github/actions/vpn-azure

./.github/workflows/sync-translation-schemas.yml -> uses: ./.github/actions/build-backend
./.github/workflows/sync-translation-schemas.yml -> uses: ./.github/actions/retry
./.github/workflows/sync-translation-schemas.yml -> uses: ./.github/actions/runner-ip
./.github/workflows/sync-translation-schemas.yml -> uses: ./.github/actions/vpn-azure

./.github/workflows/validate_resources.yml -> uses: ./.github/actions/build-vars
./.github/workflows/validate_resources.yml -> uses: ./.github/actions/retry
./.github/workflows/validate_resources.yml -> uses: ./.github/actions/vpn-azure
./.github/workflows/validate_terraform.yml -> uses: ./.github/actions/build-vars

[emvaldes]

It's impossible to determine at this point in time which ones are of value and which ones are not. This can only be defined in time when we gain a better understanding on the project and its legacy pipelines.