Data pipelines for CHAOSS metrics

[ryscheng]

Describe the feature you'd like to request

https://oss-augur.readthedocs.io/en/main/rest-api/api.html https://www.augurlabs.io/sample-page/ https://github.com/augurlabs/augur https://github.com/chaoss/augur https://github.com/chaoss/grimoirelab https://github.com/chaoss/augur-community-reports http://science.osshealth.io/repo/New%20Repositories/dataverse/overview

Describe the solution you'd like

They have an API and open source code. Their general approach seems to be around analyzing GitHub in different ways and using source code analysis.

Lots of good insights to learn from as well

Describe alternatives you've considered

We can look at other projects in the CHAOSS community

[ryscheng]

https://www.youtube.com/@CHAOSStube/videos

[ryscheng]

Here's another community project that looks really cool https://oss-compass.org/

According to some CHAOSS folks, they use GrimoireLabs on the backend

[ryscheng]

https://projects.iq.harvard.edu/osshealthindex/agenda https://projects.iq.harvard.edu/files/osshealthindex/files/2019-06-18-harrison-oss_health_index_survey_results_and_breakouts.pdf

[ryscheng]

Meeting notes:

• It's difficult to share raw GitHub data, we don't want to be violating GitHub's ToS
• We can try to run the Augur software stack separately and point it to our own Postgres database. They're interested in helping us getting this working. The tricky thing is that it's designed to run continuously as a server because they want to trickle GitHub requests according to the rate limit.
• Interested in collaborating on a data warehouse for the raw data and connecting it to Tableau or some other BI tool.

[ryscheng]

In reality, what I think we should be doing is creating dbt pipelines for CHAOSS metrics, but starting from our existing data sources