Kebechet managers metrics

xtuchyna commented 2 years ago

Is this related to another Issue?

Related to https://github.com/thoth-station/core/issues/322 The issue required to be split and refined - some of the metrics were too abstract and needed specification, other metrics may be not valuable or necessary. Also, some potential metrics that could bring value were added.

Problem statement && User stories

Following three different user stories w.r.t. to their perspectives can be identified.

As a potential future user of Kebechet,

I would like to have an overview of Kebechet managers,

so that I can

see the Kebechet's value in open source python repositories on GitHub
decide to use Kebechet in my repository ( (?) wouldn't be user prone to use only a specific manager according to the one that is used the most and has the best metrics )

As a Kebechet User,

I would like to have an overview on the use of Kebechet.

so that

(?) I can see it does not have an outage (user does not need to see this, he clearly can tell by just looking at the Kebechet behaviour in repo)
(?) I can see others use it as well (missing value here, just because others use it does not mean that I need to use it as well)
(?) overview of my own repository kebechet metrics, so that I can see it brings value to my project (this would however require comparison metrics to other projects that do not use kebechet, and we would need to show that Kebechet is better at this)

As Kebechet Maintainer,

I would like to have a look at dashboard with all Kebechet metrics

so that I can see which managers are used and performing well, which needs attention (slow responding time, critical number of rejected requests, and etc.) and which managers are not used (so even if they need attention, they can be of a low-priority).

High-level Goals

Have a dashboard regarding all Kebechet manager metrics (manager specific + SLI/SLO)
Fill kebechet universe landing page with all required data

Proposal description

Repository Daily Metrics

Collect daily metrics (that only includes Issues created in particular day) for each day of the repository history.

version manager

[x] # Issues created ~~by bot~~ by user for manager (version manager does not create issues)
[x] # Issues completed (closed by bot)
[x] # Issues rejected (closed by other)

advise manager

[x] # PRs created by bot
[ ] # PRs approved (PR contains approval OR has label approved )
- [ ] (?) extension: was labeled approved in some point of PR history
[x] # PRs merged (EDIT) extended:
- [x] merged by bot
- [x] merged by other
[x] # PRs rejected (closed) (EDIT) extended:
- [x] rejected by bot
- [x] rejected by other
[x] (?) Mean Time To Merge PR (Kebechet metrics page uses 30 days rolling window, not daily mean time) (EDIT) mean time to merge for a day in seconds

(?) other managers

Managers SLO/SLI

[x] https://github.com/thoth-station/mi/issues/572

Kebechet universe page

Satisfy metrics for the Kebechet universe page:

Kebechet is keeping (1) repositories of (2) organization on GitHub up to date. In the past month, (3) Pull Requests have been opened and (4) % of them have been merged within (5) hours. Only (6) % of pull requests were rejected or closed by humans.

[ ] (1) overall number of active (non-archived) repositories with kebechet installation
[ ] (2) overall number of unique organizations that have at least one repository with kebechet installation
[ ] (3) 30 days Rolling window sum of created PRs by bot
[ ] (4) 30 days Rolling window sum of created PRs by bot that were merged
[ ] (?) (5) Maximum Time To Merge PR value found in all merged PRs across all repositories
[ ] (?) (5) 30 days Rolling window median time of all Time To Merge attributes for all PRs across all repositories
[ ] (6) days Rolling window sum of created PRs by bot that were closed by other

Acceptance Criteria

[x] Implement daily metrics for version and advise manager
[ ] Store them on ceph as entity (for each repo under its subfolder)
[x] Decide what SLO/SLI metrics needs to be added
[x] Implement SLO/SLI metrics
[ ] Store them on ceph, specific location {namespace-name}/thoth-sli-metrics/{manager-name}/ , one file or for each day its own SLO file (?)
[ ] Make a decision on the 5. feature
[ ] Implement kebechet universe metrics
[ ] Store them on ceph, specific location (ideally under kebechet-universe subfolder) - (?) one file that is constantly overwritten or leave a dataframe of records for each day

Gkrumbach07 commented 2 years ago

/sig user-experience

Gkrumbach07 commented 2 years ago

I can see it does not have an outage (user does not need to see this, he clearly can tell by just looking at the Kebechet behaviour in repo)

The user should be able to check the status of their manager through a central support page. There may be no way to tell the user at the base of the issue.

overview of my own repository kebechet metrics, so that I can see it brings value to my project

Depending on the functionally of Kebechet we want to highlight, comparisons might be hard to measure. I.e. Time to create a tag release without Kebechet cant really be measured automatically. In UX design, number of clicks or actions to preform a task is a big metric to look out for.

xtuchyna commented 2 years ago

/assign @xtuchyna

Gkrumbach07 commented 1 year ago

Condensing EPIC due to inactivity

[ ] #603
[ ] #604
[ ] Have a dashboard regarding all Kebechet manager metrics
[ ] Fill kebechet universe landing page with all required data

thoth-station / mi