support for persistent data

[billimek]

As documented, the API contains metrics for the prior 28 days. It would be useful if there would be a way to configure this to support a 'local' json file containing longer-term data (e.g. we currently periodically merge the results into a larger json file with more historical data). Or, perhaps, support an external datasource where the API results are collected and persisted for a longer historical view.

[Myzel394]

(e.g. we currently periodically merge the results into a larger json file with more historical data

Offtopic: How do you archive the data? Do you use an open source tool or do you just call the api everyday and save its data?

[martedesco]

Thanks for your feedback @billimek

support an external datasource where the API results are collected and persisted for a longer historical view.

This is something that is being considered for a next phase.

[billimek]

Offtopic: How do you archive the data? Do you use an open source tool or do you just call the api everyday and save its data?

It's just a call to the API and merging it with an existing json file. This is how we're now handling this, via a GitHub action workflow, to override the original mocked data json as an interim way to have longer-lived persisted data:

DATE=$(date +"%Y%m%d")
curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer ${{ secrets.COPILOT_TOKEN }}" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/orgs/<our org>/copilot/usage > data_${DATE}.json
jq -s 'add | unique_by(.day) | sort_by(.day)' data_${DATE}.json src/assets/copilot_metrics_response_sample.json > data_merged_${DATE}.json && mv data_merged_${DATE}.json src/assets/copilot_metrics_response_sample.json

[andrekolodochka]

could potentially combine it with https://github.com/marketplace/actions/copilot-metrics-retention

[DevOps-zhuang]

Since current vue cli mode doesn't support save featched data to database/file, I created a nodeJS express as backend., it resue several codes from the viewer. could you pleaset take a look and check it? I am thinking what is the best way to merge it with this viewer now. same project with different folder? like below, need more advice, thanks)

https://github.com/DevOps-zhuang/copilot-metric-saver.git

project-root/ │ ├── src/ │ ├── api/ │ │ └── GitHubApi.ts │ ├── model/ │ │ └── Metrics.ts │ ├── server/ │ │ ├── server.ts │ │ ├── backend-config.ts │ │ ├── tsconfig.json │ │ └── package.json │ ├── client/ │ │ ├── main.ts │ │ ├── frontend-config.ts │ │ ├── tsconfig.json │ │ └── package.json │ └── ... │ ├── node_modules/ ├── .env

[karpikpl]

Hmm, I need to review it because I may already have another "backend" for metrics-viewer done.

we should sync on how to best combine those two

[DevOps-zhuang]

@martedesco @karpikpl I finished main featture of the backend version, it is a seperate NodeJs application. could you please check it by visitng https://github.com/DevOps-zhuang/copilot-metric-saver, thanks.

and post the highlight as below, for quick check, thanks.

Copilot Saver App Based on GitHub Copilot Usage and Seat API,provide persistent save and multi-tenant features. Since GitHubCopilot usage and seat API (see https://docs.github.com/en/rest/copilot/copilot-usage?apiVersion=2022-11-28) just return latest information. like copilot usage only returns latest 28 days usage. and manager API just return the current-latest activity. if we need a long period usage and seat for analysis. it needs a job to periodllyget such data and save it for persistent. so this project is target to meet such requirement.

This App works as backend, it will call GitHubAPI periodlly(each 12 hours by default), then save the fetched data to a file or MySQL for persistent storage, and expose some data by RestAPI, just as similar with GitHubCopilot API itself, the path is same, just update the endpoint by replacing https://api.github.com/ with http://yourserver:3000/api/

Features Fetch GitHubCopilot usage and seat data. Save data to a file or MySQL database. Analyze the saved data and expose them through RestAPI ( http://YourServer:3000/api/organization/[organizationName]/copilot/usage or http://YourServer:3000/api/enterprise/[EnterpriseName]/copilot/ )) Manage tenant information with support for organizations, teams, and enterprises. Securely handle tenant tokens and only return active tenants. call the GitHubAPI periodllyby default, it is each 12 hours. expose API description by swagger, it is http://yourserver/api-docs. so that the user can test the APIs there. Known issues For Team support in copilot user managment, since from GitHubbackend side, it only returns 'assigning_team' , not the team the user currently is in. and it needs the seat for users are assigned at team level, not individual level. we filter the team by 'assigning_team' field. Currently, there is a job in server.ts, it is fixed by below code. will set it as a parameter in future // Run job every 12 hours setInterval(runJob, 12 60 60 * 1000); for Mysql support, it use non-TSL right now. will add TSL in future.