search

codecov / engineering-team

This is a general repo to use with GH Projects
1 stars 1 forks source link

For every new org, trial bot --> and prompt to go install the app --> die after 15 days (DISCO + V0) #2142

Closed aj-codecov closed 2 months ago

aj-codecov commented 3 months ago

Problem to Solve: For most users, the largest barrier when setting up Codecov is a need to install the Codecov Github app. You cannot do this if you're not an admin in your GH org.

Proposed Solution: Enable a free Codecov bot that requires no installation for the first 2 weeks of an account that then expires. This bot should remind people to follow through on setting up the Codecov GH app for full feature usage long term. It should also enable the smoothest path to Codecov setup with minimal need for anything outside of standard setup. Said differently, this should enable full featured Codecov usage for anybody in a repo without need for admin permissions. Ideally this should work in all scenarios where the current GH app works.

Additional Info: The goal of this epic is A. Discovery and then B. an attempt to quickly build a working prototype to evaluate if our Time to first commit (TTFC) drops meaningfully after implementation. This means we'll need to be able to segment out users that use this new app and evaluate their TTFC actually does go down. If this proves successful we can move forward with total confidence that one of the largest reasons we struggle with onboarding is the barrier of GH app installation.

Adal3n3 commented 2 months ago

8/26 Making a note here: There is no design work scoped / required for this KR. If there is something needed from design probably very minimal (from @rohan-at-sentry)

rohan-at-sentry commented 2 months ago

https://github.com/codecov/engineering-team/issues/2098#issuecomment-2260969991

rohan-at-sentry commented 2 months ago

From July to today, we've had ~ 22k users sign up for Codecov. Of those 22k users, only 1700 of those selected default org. The ability to select default org is only triggered once you successfully install the Github App (or team bot). Therefore, with a "dropoff rate" of ~ 93% (this is approx, but indicative of magnitude), we can conclude that the biggest drop off in our 1st time signup experience is in the step to add a GH bot (or a team bot).

This story was supposed to add more color on where this dropoff was in a quick, experimental way, before investing in more expensive onboarding changes - with the data we've collected, I think the need to do this work is minimal.

I'm happy to close this out.

rohan-at-sentry commented 2 months ago

cc @trent-codecov @adrian-codecov