Coordinate Jenkins with GitHub Actions CI for vets-website

[U-DON]

Background

We're migrating the vets-website Jenkins/Circle pipelines to GitHub Actions and will start by moving some steps, such as unit tests and linting.

There are other steps like the content build and deploy that will take more time and discovery to migrate to GHA. Those will have to remain in Jenkins while we migrate the more straightforward steps over to the GHA pipeline.

Currently, dev and staging deploys occur at the end of the Jenkins pipeline only in the master branch. As we're migrating tests and such to GHA, we'd like for successful runs of the GHA pipeline to trigger the deploys in Jenkins.

Approach

To split the pipeline between GHA and Jenkins, we could configure a webhook in the GitHub repo to communicate the success of a GHA workflow to trigger the rest of the steps (including deploys) in Jenkins.

• It should be invoked in response to successful runs of the build and test GHA pipeline (to be created) on master.
• It should run the Jenkins deploys to dev and staging (deploys/vets-website-vagovdev and deploys/vets-website-vagovstaging).
• Prod deploys are handled by the daily autodeploy (deploys/vets-website-auto-deploy), but the build status it checks for might need to be updated to be based on or include the GHA pipeline.

We considered modifying the dev/staging deploy step to poll for the status of the GHA run for the same commit via GitHub API, but the webhook approach is preferable to this.

• Unnecessarily extends the Jenkins pipeline runtime.
• Less reliable and inconsistent with timing than a webhook.
• Involves more work in Jenkins even though we're trying to move away from maintaining it.
• Should only be pursued if the webhook doesn't work.

[dginther]

In order to allow GHA to trigger a workflow in Jenkins, I believe the following is needed:

• custom webhook receiver in Jenkins
• make sure revproxy will allow the new webhook / route the new webhook

[omgitsbillryan]

There's tool on the GH Actions marketplace called Trigger Jenkins Job. It requires that a "user token" be set on the jenkins user, which in this case would probably be the va-vfs-bot. Once configured we could configure the GHA like so:

    - name: Trigger jenkins vets-api deploy
      uses: appleboy/jenkins-action@master
      with:
        url: "http://dev.va.com/<some_path>"
        user: "va-vfs-bot"
        token: ${{ secrets.TOKEN }}
        job: "deploys/vets-api-server-vagov-dev"

I'm concerned about security, in that, using scripted user authentication with the va-vfs-bot is over privileged. I think this can be mitigated in the revproxy by using a static route.

[omgitsbillryan]

I think another option would be to use the generic-webhook-trigger jenkins plugin in conjunction with github-action-for-curl.

[U-DON]

Imagine the dashed line moving to the right as the steps get moved to GitHub Actions.

TLDR: Carry on and work on configuring GitHub Actions to kick off the Jenkins job.

--

Initially, I had envisioned the possibility of having the GitHub Actions and Jenkins parts (minus deploy) of the pipelines run in parallel before kicking off the deploy part (in Jenkins; represented by the blue box) via webhook.

After thinking about it some more, it would probably be fine to simply run the GitHub Actions pipeline followed by the rest of the Jenkins pipeline in series. It would be equivalent to the behavior of the current Jenkins pipeline. If it's simpler (which it sounds like it is) to (A) just invoke the Jenkins job from the GHA pipeline than (B) coordinating two parallel pipelines to trigger the webhook to do the deploy, then the former (A) would be the better approach.

[U-DON]

For reference, the GHA pipeline is WIP in department-of-veterans-affairs/vets-website#16516.

[jhouse-solvd]

@meganhkelley and/or @U-DON - would you consider this "In progress"?

[U-DON]

@jhouse-solvd: Just saw that @omgitsbillryan's PR has merged (thank you!). I would say this is still in progress, but I think we might need the user and API token before we can use those endpoints with the jenkins-action job.

[omgitsbillryan]

Agreed. The current plan is to re-use the va-cms-bot for now, and generate a new token on that jenkins user. Once generated, that token will need to be stored as a Github Secret. Once the secret exists, we can then have a code snipped like:

- name: trigger single Job
  uses: appleboy/jenkins-action@master
  with:
    url: "http://dev.va.gov/jenkins"
    user: "va-cms-bot"
    token: ${{ secrets.API_TOKEN }}
    job: "job/deploys/job/vets-website-vagovdev/build"

As an added wrinkle, api tokens are not persisted across jenkins deploys and must be manually regenerated each time. I'm coordinating with @tskinn on how/if/when this will be addressed since it affects this work.

[omgitsbillryan]

I was able to get a curl request through to kick off a job:

curl -X POST -L --user va-cms-bot:<API_TOKEN> https://dev.va.gov/jenkins/job/deploys/job/vets-website-vagovstaging/build --data-urlencode json='{"parameter": [{"name":"notify_slack","value":true},{"name":"ref","value":"cd587193ea7a8685d917b0a0443bb75a2a418cea"},{"name":"app","value":"vets-website"},{"name":"environment","value":"vagovstaging"},{"name":"migration_status","value":false}]}'

I had originally intended on using https://github.com/marketplace/actions/trigger-jenkins-job but realized that it doesn't support sending job params to the job. I think I'll end up using https://github.com/marketplace/actions/github-action-for-curl instead. If we want a more formal solution, I bet it wouldn't be too hard to create our own GHA module for improved readability and to encapsulate this function in one code base rather than having curl commands running in all the various GHA workflows.

[U-DON]

Looking it over again, we might actually need the testing/vets-website jobs for now rather than the deploys. Apologies for the confusion! The idea is that we'd truncate stuff from the start of that pipeline as we move them to GHA.

And there are a few more steps that we haven't accounted for in the GHA workflow yet such as the archiving and review instance steps. We're in the midst of figuring out the build step as well.

But we did have one concern if we were to kick off testing/vets-website. Since there's a webhook for Jenkins to run that build automatically, I'd be concerned about it running twice (once from webhook and again from the GHA workflow calling it). Would it be possible to disable it for testing in a branch so it doesn't duplicate itself?

[omgitsbillryan]

I poked around the config and it looks like there's support for excluding certain branches by name - is this what you want?

testing/vets-website is a "multibranch pipeline project". It discovers new branches and creates new "jobs" for each one. I suspect that filtering branches by name may mean that no job would exist at all for your branch. Specifically, http://jenkins.vfs.va.gov/job/testing/job/vets-website/job/YOUR_BRANCH would not exist, and such, http://jenkins.vfs.va.gov/job/testing/job/vets-website/job/YOUR_BRANCH/build would also not exist.

[omgitsbillryan]

Here's a total hackjob of a solution - add a param (e.g. "cancelBuild") to the job that auto-aborts the build by default. This way MBPJ still auto-detects and creates the job but it gets immediately aborted. Then from GHA, you can trigger said job using cancelBuild=false. I whipped up a small POC to see that it works - link.

[U-DON]

Hacky but effective! I think that could work.

• If GHA calls the job, would it overwrite the aborted status for the Jenkins build in the list of checks?
• Could we add /jobs/testing/vets-website to the list of allowed_api_endpoints?

[jhouse-solvd]

@U-DON and or @omgitsbillryan - could you add a comment with the summary of the current status of this work? thank you.

[U-DON]

The necessary API endpoints (testing/vets-website and testing/content-build) have been exposed for the GitHub Actions workflows to use. Confirmed to be working from a successful run in a branch in the content-build repo.

Keeping this open for FE Tools to track remaining work until the combined GHA + Jenkins workflow is running in the master branch of both repos.

[jhouse-solvd]

Removed the operations label for now, but can always re-add if anything is needed.

[oseasmoran73]

Closing ticket as PR has been recently merged to trigger Review Instances via Jenkins from Github Actions