[13pt] Create reports out of prescriptions

fridex commented 3 years ago

Is your feature request related to a problem? Please describe.

As Thoth administrator, I would like to know statistics about prescriptions - the open-source database for Python projects. To support this, I would like to have periodic reports that show insights about the database.

Describe the solution you'd like

Create a periodic job that introspects thoth-station/prescriptions and aggregates relevant information about the open-source database about Python projects. A very first implementation can give insights about:

[ ] how many prescriptions are available in the database
[ ] how many projects are available in the database
[ ] types of pipeline units that are available in the database and their count
[ ] TOP packages with respect to the number of prescriptions available
[ ] TOP packages with respect to contributors that add new prescriptions
[ ] TOP changes since last week in prescription available (or another period of time)
[ ] added new projects since last week (or another period of time)
[ ] new contributors to the database
[ ] number of changes done since last week (or another period of time)
[ ] stats on specific prescriptions count (ex. number of scorecard specific prescriptions, number of prescriptions that show GitHub link, ...)
[ ] create and host a dashboard on operate-first

Later, we can extend the stats to cover also information about Python package indexes, static source code analysis and its use and such.

goern commented 3 years ago

is that something that we want to show in a grafana dashboard?

fridex commented 3 years ago

is that something that we want to show in a grafana dashboard?

I don't know what presentation layer would suite this the most - might be grafana dashboards or an e-mail report. @pacospace will know more.

goern commented 3 years ago

let's start with a dashboard.

/sig observability /priority important-soon /triage accepted

sesheta commented 2 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

fridex commented 2 years ago

/remove-lifecycle stale

sesheta commented 2 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

harshad16 commented 2 years ago

/lifecycle frozen

VannTen commented 2 years ago

Most of these should be easily doable by cloning the repo and doing some find tricks:

how many prescriptions: find -name '*.yaml'
how many project: find -type d -mindepth 2 -maxdepth 2
types of pipelines and their count: find -name '*.yaml' -exec grep 'type:' {} + | cut -d : -f 3 | sort | uniq -c
top packages by prescriptions count : find -name '*.yaml' -printf '%h\n' | rev | cut -d / -f 1 | rev | sort | uniq -c

If we go the metrics routes (promql ones), most "change" metrics should not be added as is, but derived from counter / gauge.

harshad16 commented 2 years ago

Suggestions:

curate the metrics in fashion of metric-exporter.
Have this as a part SLO-reporter or Dashboard or on Website.

Acceptance criteria:

[ ] Explore the various metrics to be created.
[ ] Create them via metric-exporter to expose to prom.
[ ] include it in slo-reporter or a dashboard whichever is convienent.
[ ] include a description on website, so users can view these information.

xtuchyna commented 2 years ago

So there is no need to create metrics workflows for this inside mi then, @VannTen @harshad16 ?

VannTen commented 2 years ago

I don't know what mi does exactly, so I'm not in the best position to answer. But if we're talking about prescriptions and agree that thoth-station/prescriptions is the canonical source, yeah, there should not be metrics elsewhere on the same data.

harshad16 commented 2 years ago

We wanted to explore the front of metrics and see if that would be suitable for this. With MI can go do Prometheus metrics?

xtuchyna commented 2 years ago

Hey @harshad16 , sorry to bother but not sure what you meant by

With MI can go do Prometheus metrics?

Please let me know! thxx

harshad16 commented 2 years ago

ah typo, i meant can we do the Prometheus metrics with MI for the details insight list in the description?

thoth-station / thoth-application

[13pt] Create reports out of prescriptions #1949