ccerv1
commented
3 weeks ago Notebook:
- https://github.com/opensource-observer/insights/blob/main/analysis/thank_arb/thank-arb-starter.ipynb
Outline: How to Set Up and Analyze Data in a Notebook with Open Source Observer Data
-
Getting Started
- Introduction to creating an analysis notebook with open source observer data.
- Using an existing collection as an example due to the new collection ThankARB not yet existing.
-
Setting Up Connection
- Steps to connect to Google Cloud and BigQuery for data access.
- Options: Use Jupyter notebook locally or Google Colab for cloud-based access.
- Demonstration of connecting by storing Google credentials locally.
-
Configurations
- Creating basic configurations for the project and dataset.
- Recommendation to start with the OSO Playground dataset for exploration to avoid excess data scanning costs.
-
Query Testing
- Testing queries directly in BigQuery SQL Explorer.
- Example query on the OSO artifacts table and the OSO Playground dataset for comparison.
-
Data Filtering
- Filtering artifacts by project and project collections.
- Filtering artifact sources, e.g., GitHub, to narrow down the data.
-
Activity Metrics
- Converting list of project IDs into arguments for SQL queries.
- Retrieving code metrics for projects in the list.
-
Timescale Metrics
- Constructing a query to retrieve time series metrics data.
- Joining metrics data with artifact names for better understanding.
-
Data Visualization
- Generating tables and charts from the data.
- Creating a project snapshot table and an area chart for time series metrics using Plotly.
-
Exporting Results
- Generating a markdown export for creating a report with a formatted table.
- Saving the analysis in a repo for future reference.
-
Conclusion
- Recap of the process and the availability of the analysis in the specified repo.
- Thanking the audience for their attention.
Link to Loom
What is it?
Help @rohitmalekar get going with querying OSO data and creating some initial analysis.