Cody metrics

[malomarrec]

Cody events

• Login flow
  • CodyVSCodeExtension:acceptTerms:clicked
  • CodyVSCodeExtension:login:clicked
• Recipes used from menu
  • CodyVSCodeExtension:recipeExplainCodeDetailed:clicked
  • CodyVSCodeExtension:recipeExplainCodeHighLevel:clicked
  • CodyVSCodeExtension:recipeGenerateCode:clicked
  • etc for all recipes
• Ask Cody right click menu
  • CodyVSCodeExtension:askCodyExplainCode:clicked
  • etc for all cody recipes
• CodyVSCodeExtension:question:submitted when a generic question is submitted
• CodyVSCodeExtension:startNewConversation:clicked when a generic question is submitted
• CodyVSCodeExtension:openMenu:clicked
• CodyVSCodeExtension:about:clicked
• CodyVSCodeExtension:logout:clicked

Aggregated pings

• A DAU is someone that has performed any non-login, logout, or view about event in a given day
• activation is TBD, need a bit of amplitude data to see what's a good predictor of activation.

Attributes

• codyIsLoggedIn: true if the user is logged in to a Sourcegraph instance, false else

Site admin attributes:

• codyCompletionsEnabled: true if set, false if not
• codyCompletionsModel: the name of the model in embeddings.completions
• codyEmbeddingsEnabled: true if set, false if not
• codyEmbeddingsModel: the name of the model in embeddings.model

[nathan-downs]

Recipes used from menu events: CodyVSCodeExtension:recipeGenerateCode:clicked CodyVSCodeExtension:recipeGenerateUnitTest:clicked CodyVSCodeExtension:recipeGenerateDocstring:clicked CodyVSCodeExtension:recipeImproveVariableNames:clicked CodyVSCodeExtension:recipeTranslateToLanguage:clicked CodyVSCodeExtension:recipeGitHistoryt:clicked

Ask Cody right click menu events: CodyVSCodeExtension:codyToggleEnabled:clicked CodyVSCodeExtension:askCodyExplainCode:clicked CodyVSCodeExtension:askCodyExplainCodeHighLevel:clicked CodyVSCodeExtension:askCodyGenerateUnitTest:clicked CodyVSCodeExtension:askCodyGenerateDocstring:clicked CodyVSCodeExtension:askCodyTranslateToLanguage:clicked CodyVSCodeExtension:askCodyGitHistory:clicked CodyVSCodeExtension:askCodyImproveVariableNames:clicked CodyVSCodeExtension:codySetAccessToken:clicked CodyVSCodeExtension:codyDeleteAccessToken:clicked CodyVSCodeExtension:askCodyViewSuggestions:clicked

[nathan-downs]

Per Slack thread with @malomarrec we will use the CodyVSCodeExtension:acceptTerms:clicked event to determine when access has been provisioned to a user.

[nathan-downs]

• [x] Stage 1: Get the EventLogger integrated into the cody-shared library and send all event level telemetry to sourcegraph.com only (we won't be able to differentiate the company users are from initially).
• [x] Stage 2: Add event level telemetry to the local instance Postgres database for aggregation via pings
  • [ ] Add Cody events to admin analytics page
  • [ ] Add Cody to pings

[dadlerj]

Just FYI, I didn't mean to imply in that thread that we need to go add custom dashboards to the admin analytics page (though the IPA team might want to!). I mostly meant that the admin wouldn't see that Cody user as a DAU, WAU, MAU until we reached stage 2. So without doing anything new, the admin analytics dashboard should start to show higher usage numbers once you get the first bullet on stage 2 complete.

[nathan-downs]

Got it. For stage 2 I was thinking along the lines of having it roll up into the ExtensionUsageStatistics as a separate ExtensionID, though looking at it briefly it doesn't look like much would need to be changed on the backend.

[akalia25]

@nathan-downs this PR logs the events we want once eventLogger is added to Cody.

• https://github.com/sourcegraph/sourcegraph/pull/50139

[nathan-downs]

This PR is passing CI and is ready to merge if we want logging of these events immediately to the connected Sourcegraph instance only:

CodyVSCodeExtension:codyToggleEnabled:clicked CodyVSCodeExtension:codySetAccessToken:clicked CodyVSCodeExtension:codyDeleteAccessToken:clicked

CodyVSCodeExtension:askCodyExplainCode:clicked CodyVSCodeExtension:askCodyExplainCodeHighLevel:clicked CodyVSCodeExtension:askCodyGenerateUnitTest:clicked CodyVSCodeExtension:askCodyGenerateDocstring:clicked CodyVSCodeExtension:askCodyTranslateToLanguage:clicked CodyVSCodeExtension:askCodyGitHistory:clicked CodyVSCodeExtension:askCodyImproveVariableNames:clicked

We'll have the replicated events sending to the dot-com endpoint later today, which will come into BigQuery in the telligentsourcegraph.dotcom_events.events table from the event source CODY thanks to this change that was merged today. Currently working on the dot-com event firing and the rest of the events now.

[nathan-downs]

https://github.com/sourcegraph/sourcegraph/pull/50144 just needs a stamp from someone on the Cody team before merging but is passing CI and our manual testing. It satisfies stage 1 and 2 of previous comments, but there is still another PR needed for adding the events to pings for private instances (they'll just be logged in the local instance event_logs table) and the events will need to be added to the allowlist for managed instances. As it stands, we'll at least be getting events coming through to the dotcom endpoint. Example, the first screenshot shows the event in the local instance event log (in this case our s2 instance) and the second screenshot shows that same event coming into the dotcom events table in BigQuery as an anonymous event. We're passing the originating endpoint in the public argument, so combined with the anonymous user id we'll be able to aggregate basic statistics from these anonymous events. In cases where users are logged into dotcom in the Cody extension we bypass the anonymous request and are able to pass user id, feature flags, and other metrics available from dotcom.

[nathan-downs]

Re: the pings addition, I'm imagining something like a CodyUsageStatistics akin to how some our other metrics are gathered (e.g., SearchUsageStatistics).

[malomarrec]

@nathan-downs is there documentation for follow-ups here? Such as getting events through pings on customer instances?

[nathan-downs]

Not yet, I can create a ticket for getting that added to pings, but it won't go out until the next Sourcegraph release (or patch release if we need it out sooner)

[malomarrec]

Yup, I'd just like to get it tracked. @beyang could we get this added to the next patch release? Or is it against release guidelines?