Police-Data-Accessibility-Project / meta

Planning our activities with issues that don't fit in a specific repository yet.

GNU General Public License v3.0

693 stars 58 forks source link

Centralized Metrics view #169

Closed josh-chamberlain closed 1 year ago

josh-chamberlain commented 3 years ago

We're considering using the Notion API or Django here. Notion is configurable without code, and one less app to deploy. Django is more customizable.

GitHub

[ ] Police-Data-Accessibility-Project/PDAP.io#41
[ ] Numbers of scrapers which exist
[ ] Percentage of scrapers working

DoltHub

Dolt SQL API: https://docs.dolthub.com/dolthub/api

[ ] PRs merged in the DoltHub repo (by user) helpful blog post
[ ] Number of Datasets documented / Estimated total
[ ] Number Agencies documented / Estimated total
[ ] Number of agencies with all known datasets scraped

milespratt commented 3 years ago

I can start scoping this out. How is each metric gathered now? Are they regularly published anywhere else? How often do you want them updated on the site?

josh-chamberlain commented 3 years ago

@milespratt those metrics were gathered once, manually, and added to the deck.

PRs merged → GitHub API? Community Members Introduced → new people in Discord? Datasets Scraped → this one's tough, but super important. Datasets Documented → this one's easy enough to glean from our datasets repo and tables therein, particularly "Agencies" and "Datasets": https://www.dolthub.com/repositories/pdap/datasets

So maybe a better version of this issue would recommend that we could use a proof of concept for getting a simple metric from the Dolt API to our site: https://docs.dolthub.com/dolthub/api

DoltHub is where we'll be making the most progress in the near term in terms of experiments and bounties.

Other ideas for metrics:

datasets added
agencies updated
datasets covering n states
data intake for n agencies (we only have a few examples of data-intake so far)

CaptainStabs commented 3 years ago

I can start scoping this out. How is each metric gathered now? Are they regularly published anywhere else? How often do you want them updated on the site?

We can get a (rough) dataset count by counting python scripts in the scraper's repo excluding config related files of course.

As for update intervals, we can just run it on merge

CaptainStabs commented 3 years ago

Community Members Introduced → new people in Discord?

Discord supports widgets for websites. See their blog post on it here

PRs merged → GitHub API?

GitHub also has webhooks for this if it makes this any easier. GitHub webhooks