2i2c-org / infrastructure

Infrastructure for configuring and deploying our community JupyterHubs.
https://infrastructure.2i2c.org
BSD 3-Clause "New" or "Revised" License
106 stars 65 forks source link

[Platform Initiative] Hub Scale Cost Monitoring for GCP #5063

Open Gman0909 opened 1 week ago

Gman0909 commented 1 week ago

Productboard link: https://2i2c.productboard.com/roadmap/7947557-2i2c-roadmap/features/26823459

Description

Institutional leads, as well as department leads in large organizations, need to be able to justify their budgets and ensure they are being spent with value in mind. Business intelligence depends on data, and we want to make sure we build towards a data reporting infrastructure that can make a hub or constellation of hubs' usage and cost more transparent, enabling better decision making come budget time, as well as offering a sense of security and transparency over a service that is often perceived as being a high risk for cost overruns.

To that end, we would like to give community and institutional leaders the ability to monitor the cost and usage of a hub, or groups of hubs, provided by 2i2c.

The solution should provide a dashboard that automatically updates to reflect up to date aggregated costs and usage reports for each hub in a constellation, or the single hub an administrator has admin rights over. Data should be able to be exportable in the form of reports.

Additionally, we should investigate adding an option to share the dashboard with individuals outside of those with administrative privileges.

Typical use cases:

Scope

We already have a document listing the things cloud providers charge you for. The things we care about, in priority order, are:

  1. Home directory storage (NFS)
  2. Object storage (scratch and persistent)
  3. Compute (nodes)

Each of these costs should be attributable to either:

  1. A common pool that serves all of the hubs (primarily, the core node pool and staging hubs)
  2. A particular hub

Attributing to individual users, or specific subgroups inside a hub, are out of scope.

Definition of Done

Admins of any 2i2c hub can access dashboards and reports where they can monitor up-to-date cost information for their hubs, and export reports with that same information.

### Tasks
- [ ] https://github.com/2i2c-org/infrastructure/issues/4474
- [ ] https://github.com/2i2c-org/infrastructure/issues/4551