Investigate AI Review for our PRs

[jimleroyer]

Description

As a developer, I want to have my code reviewed by AI, So that it can catches common errors that humans might overlook once in a while Such sensitive Terraform variables not being flagged as such in the code changes.

WHY are we building?

This is following an incident action items as we did not catch a sensitive variable and that caused a leak through our logs. We want to catch that sort of bugs as early as possible in the development pipeline. An AI could catch these sort of easy to spot (but potentially easy to miss) changes and alert us on the pull requests.

WHAT are we building?

Integrating with an AI that reviews our pull requests. Namely, this one was spotted and could be tried: https://github.com/marketplace/actions/ai-code-review-action

VALUE created by our solution

More security for this specific use case, but also higher code quality in general by an AI bot reviewing our code.

This might mean some noises too as well on code changes that the AI bit might be incorrect or unaware of context.

Acceptance Criteria

• [x] A prototype of the AI code review integration is presented to the GCNotify core team.
• [ ] A review document is provided with a summary of the capabilities of the code review AI. This should be tested against some common scenarios such as introducing a password variable in the Terraform repository which should be tagged as sensitive but is not.

QA Steps

• [x] The PR review bot is able to deliver reviews on PR.
• [ ] We have a document for hosting results of the review quality.
• [ ] We have a system in place to collect the quality data on the reviews.
• [x] This should be tested against some common scenarios such as introducing a password variable in the Terraform repository which should be tagged as sensitive but is not.
• [x] Tested against a few PRs in
  • [x] Terraform,
  • [x] manifests.

[P0NDER0SA]

We need to get an OpenAI API key

[P0NDER0SA]

https://github.com/cds-snc/notification-api/pull/2254

[ben851]

@ben851 to QA

[ben851]

Waiting on Calvin to give us a new API key so we can re-enable the PR bot.

[ben851]

We received the API key, and merged the new job and trigger. Needs to go back onto terraform

[sastels]

Steve will try on one of his terraform PRs

[sastels]

looked good against my terraform PR! Haven't tried a manifest one yet though.

[sastels]

Steve will make a PR to add a fake password or something that's not marked sensitive.

[sastels]

AI said variable critical_government_password should be marked sensitive and did not say that variable google_url should be sensitive ✅

[sastels]

did a manifest PR but it was just changing a variable - AI review ran but had nothing t0 say. We need to run with a more substantive manifest pr...

[P0NDER0SA]

We might open this up to some other repositories to get more feedback because it's not costing a lot of money so far!

[P0NDER0SA]

We are going to run this until Xmas and then review the costs and benefits for this and make a decision!

[P0NDER0SA]

going to add a slack reminder for sometime around xmas

[P0NDER0SA]

I added a slack reminder for December 20th