AI Code review and test generation GA plan

[rohan-at-sentry]

For Q4: EA for AI features will hit mid November. We are transferring AI PR Review over to Seer in the short term. This is step 1. UI work should be minimal throughout the quarter, but we will continue to fine tune prompts, respond to feedback and improve what we have currently built. There's two areas of interest for discovery:

1. Onboarding - how do we make this as easy as possible?
2. Given the sharing of resources between Codecov and Sentry to power this feature, the optimal group of users to target is our Sentry plan users, who should have the easiest path to usage and value with these features - this is worthy of design discovery during this quarter.

[eliatcodecov]

I'll provide more clarity on this one then pass off to @aj-codecov :

Context

Before getting into the specifics, it's important to remember that there are two AI apps currently in play within Sentry and we'll be working to consolidate those over the coming weeks. The apps:

• codecov/ai-review-prompt -- this is a laravel implementation that supports the Codecov AI Reviewer github app integration. It communicates directly with OpenAI to provide AI PR Review and doesn't use any Sentry internals to do this. It also provides AI Test Generation (but there are some issues / work to be done to get this fully working, see Porting into Seer section below)
• getsentry/seer -- Sentry's AI/ML platform. The intent of this project is to provide an API to services within Codecov/Sentry to easily interact with various AI/ML models and agents. Currently AI Test Generation is implemented in seer and codecov/ai-review-prompt uses this functionality via REST API. However, this API is not live in production, therefore the codecov/ai-review-prompt cannot provide test generation in a production capacity at the moment.

Onboarding

So there are two apps, and for a short period of time AI PR Review will be offered solely in one (codecov/ai-review-prompt) and Test Generation solely in the other (seer).

Since we will be offering AI PR Review via our own integration, we need a consent flow built into the Codecov UI for use with our Codecov AI Reviewer github app integration. @Adal3n3 already has designs for this, they've been 👍 by legal/compliance and I think some amount of implementation work even happened last quarter, but I could be wrong.

⚠ It's very important that we ship this consent/onboarding flow before November 19th. ⚠

Porting into Seer

Currently AI Test Generation is only available in Seer. It will remain that way until Seer provides a production API such that Codecov's implementation can rely on Seer for test generation results. The most important thing we can do here is lend support to Sentry's AI/ML team if it is needed to help get this across the finish line. Given his knowledge of the system @rohitvinnakota-codecov should likely take point on that.

@rohitvinnakota-codecov will also be porting AI Code Review to Seer such that it can be used from the @sentry copilot extension. Once this is done, codecov/ai-review-prompt will forego its own direct integration with OpenAI and use Seer for AI PR Review via REST API (or some other mechanism) instead.

It's crucial we deliver the above three things this quarter. Specifically:

• A consent flow in Codecov's UI. (must happen by Nov 19th)
• A port of AI Code Review to Seer and the subsequent update to codecov/ai-review-prompt to use Seer's implementation of AI PR Review instead of its own (should happen by EoQ)
• Some sort of communication mechanism between Seer and codecov/ai-review-prompt in production. The best we can do here is play a supporting role if needed.

Longer Term

Longer term, we should move away from codecov/ai-review-prompt. It was never really meant to be a long term production app and contains a lot of code and features that aren't necessary long term. Additionally, it's a laravel app, and I think most folks on the engineering team would prefer something in Python for consistency with our other products. However, this app can get us by for quite some time while such a migration happens. @rohitvinnakota-codecov has the most context on this aspect of the project, and can provide more detail to engineering if needed.

Resources needed / Direct Asks

• @Adal3n3 to review consent flow designs in Codecov UI and reconnect with engineering to follow up on this work and determine if any additional work needs to be done.
• Applications team to implement consent flow in Codecov UI (please feature flag until Nov 19th)
• @rohitvinnakota-codecov to work on port of AI PR Review into Seer
• @aj-codecov to provide product and project management support to this initiative.