search

codecov / engineering-team

This is a general repo to use with GH Projects
1 stars 1 forks source link

Discovery 1: GitHub Action Experience #2093

Open codecovdesign opened 3 months ago

codecovdesign commented 3 months ago

follow up from kickoff to discovery on 7/16, per epic: https://github.com/codecov/engineering-team/issues/2067

Context and overview:

We are exploring/hypothesizing that by leveraging the existing action and potentially integrating with the CLI, we could create a more seamless experience without the GitHub app, then transitioning to better incentive user toward GitHub app installation.

Discovery

Ideation: modify the existing GitHub Action with CLI support to operate independently of the GitHub app.

Modify existing action Leverage CLI Revise action to function without the GitHub app Determine how to tie reports to the app and display them effectively

steps to explore:

Metric or success criteria:

codecovdesign commented 3 months ago
sync notes on 7/30
Scenario: assuming there was a modified action that didn't require secret or GitHub app (as [seen here](https://file.notion.so/f/f/bbfa457d-be3e-4cea-ac9c-9aff172f04f1/7b20de8e-7bd1-436a-956a-35fca97fe2dd/App-less_Codecov.pdf?id=04076644-be65-4554-97b9-3e8558f9892b&table=block&spaceId=bbfa457d-be3e-4cea-ac9c-9aff172f04f1&expirationTimestamp=1722356622129&signature=dh9tb4y2X5peFrKXSqBxUZcVak2s_PPQJFSGCePWz4U&downloadName=App-less+Codecov.pdf)): - What would be provide to user, in terms of reporting? - How does the modified CLI usage impact the flow? (outside of the scope of GitHub action) - What would/could the system know (repo, org) that is using action?
- scenario you apply a codecov action W/O token - we could explore this - one way we were thinking about - appless definition? - 1. ux involves the app for configuration - 2. ux doesn't involve the app in any way (GH action) - do we want an action that doesn't require - alternative token input - once provided the token it would need to be as a parameter - add issue to anchor example repo explore demo concept todos - clarify IF we are also interested in a tokenless GH action - @codecovdesign to pack up design concepts + API questions + sprint planning for discovery
codecovdesign commented 3 months ago
action details/exploration goals/notes point of clarity, per review with @jerrodcodecov : - the exploration of this modified action should look at offering an action that that can operate standalone (ex: w/o token or Codecov login for configuration). The question then following this is what could be displayed/offered with this action. the goal is OSS contribution in turn to have more adoption and familiarity with the Codecov product. - @eliatcodecov : there is a distinctively different action than today, the action shouldn't require anything from Codecov in this scenario. It could be that they use the existing one today, but there is an option that doesn't walk to Codecov at all.
codecovdesign commented 3 months ago

@matt-codecov @michelletran-codecov assigned this to you tentatively, nothing is needed at this time, but for review and I'm setting up a kickoff sync to review/discuss further for next steps. per review with @trent-codecov

notes from aug 5th sync - goals for discovery: what could this action provide if it never left and/or required not having any relationship to the codecov? - directions to consider: 1) how do we provide maximum value that is mostly standalone, 2) how do we provide minimum bar of value that helps adoption to get users started w/o having to do codecov.io flow as-is today. - summary: an offering that is standalone action with goal of offering a lower friction entry point that offers some codecov value, with business goal for further adoptions - what could we provide if action didnt' talk to codecov at all - if running all tests would it be project coverage? or patch coverage - could do either through git diff; though the limiting factor is customers that have multiple CI jobs - do we advise that action is put at the end of CI and processed locally - then question is from this could we calculate patch coverage - tokens to consider: - 1) codecov UI token that helps authenticate (static but can be regenerated in ui), - 2) when user install GH app - GH then provides a token we can use to act on behalf of the user (allows us to write a comment as well as obtaining commit/pr information). We get a different token every time, but it's something we have all information to regenerate - 3) user's GH action instance is given a token by GH that allows GH action to make certain requests to GH - idea being that there might be logic and/or ability to tests results where there is no app to install but it needs to be supported (typically short lived). - If we wanted to leverage this we'd need GH to send every time OR we may not be able to leverage - 4) the initial sign in of user authentication; this authentication then gives us a token to behave on behalf of the username. IF token 3 was able to be leveraged and the data to be processed, then in theory we could use the data and the user could see it in their UI. main question is could we use token 3 temporarily - could it be a ToS violation or not feasible. - what are next steps for investigation - a bit of market research? - on implementation details side, would be helpful to know if we can use some caching action and/or if we can make multiple actions runs and a finisher to tie it all together - would we be shipping language parsers? would we need to have a smaller version than the python report? - we probably want to support all parsers we have - yes, that fine to have a smaller/dumber version of the python report, but unsure - re: the rust piece, is it possible to ship with rust included? already has bindings planned - hard to see how much work is involved in completing this - most work appears to refactoring or at least getting rust further along to leverage it. seems like a lot more research, maybe not that much glue. - better to refactor python or putting more resources on rust side? - overall would be ideal to have some plan forward at end of quarter