search

Green-Software-Foundation / hack

Carbon Hack 24 - The annual hackathon from the Green Software Foundation
https://grnsft.org/hack/github
14 stars 1 forks source link

GCP Importer #96

Open tianbiao opened 2 months ago

tianbiao commented 2 months ago

Prize category

Best Plugin

Overview

Story

As a user of a Google Cloud Platform, I want to be able to retrieve usage metrics for my application. Right now, IF only enables me to do this for Azure VMs.

Rationale

We have implemented an Azure importer in IF, but no such model exists for Google Cloud Platform. This will address the current limitation of IF having a Azure-only importer and enable GCP users to retrieve observations of GCP resources via Google Monitor (or similar APIs) and pipeline that utilisation data into other IF models. As suggested here (https://github.com/Green-Software-Foundation/hack/discussions/30)

Scope

This should be a model plugin that conforms to the IF model plugin interface.

Implementation guidelines

This model MUST conform to the IF model plugin interface The model MUST return metadata about the application and usage data (e.g. CPU-util, RAM-used, etc) similar to the azure importer The model must return data in the expected units (kWh, seconds, GB...) The model must accept duration in units of seconds an input. The model SHOULD be written in Typescript The model MUST have unit tests that demonstrate correct execution The model MUST have documentation in the form of a README that explains how to use.

Questions to be answered

No

Have you got a project team yet?

I am working on this project alone

Project team

No response

Terms of Participation

Project Submission

Summary

As a user of Google Cloud Platform, I want to be able to retrieve usage metrics for my application. GCP importer Plugin enables GCP users to retrieve observations of GCP resources via Google Monitor APIs and automatically populate your manifest with usage metrics that can then be passed along a plugin pipeline to calculate energy and carbon impacts.

Problems

To enable GCP users to retrieve observations of GCP resources, below problems has been addressed

  1. Authentication with GCP APIs
  2. Retrieve the monitor metrics of Compute engine VMs from all regions of one GCP projects, eg: CPU and memory usage
  3. Populate usage metrics that can be passed along with other IF plugins

Application

Prize category

Best Plugin

Judging criteria

Overall Impact

GCP is the world's Top3 cloud provide, the GCP importer plugin can enable GCP uses to use the Impact Framework

Opportunity

The GCP importer is easy to use for systems hosted in Google Compute engine VMs. And the GCP importer plugin is easy to be extended to support more GCP services such as App engine, Cloud Run

Modular

The plugin is built based on the arch of Azure importer, and it works well with other plugins such as CCF

Video

https://youtu.be/q4mPejLqj-A

Artefacts

https://github.com/tianbiao/if-unofficial-plugins/tree/gcp-importer

Usage

https://github.com/tianbiao/if-unofficial-plugins/blob/gcp-importer/src/lib/gcp-importer/README.md

Process

Inspiration

  1. I am a GCP user for 4 years
  2. I'd like to understand and contribute to Green Software open source tools.

Challenges

  1. Choosing the right Google monitor metrics when retrieving CPU and memory utilizations
  2. As the GCP provides a lot of compute services, for example Compute engine, App engine and Kubernetes Engine, it's hard to implement the GCP importer plugin to support them all during such short hack time.Finally, I start to build the plugin from Compute engine usage metrics.

Accomplishments

The GCP importer plugin can response enough details of the usage data for Carbon calculations.

Learnings

What's next?

  1. Introduce to my teams that is using GCP to use IF and this plugin to calculate Carbon
  2. Extend the current plugin to support more GCP resources
jmcook1186 commented 2 months ago

Great - this would definitely be a useful plugin! Have you looked into what usage metrics are available from GCP APIs?

jawache commented 2 months ago

Agreed very excited about this plugin!

One thing I might suggest in terms of approach is to have the plugin return data as a flat set of observations and then allow users to use the group-by plugin to group the data how they want, see this as an example: https://github.com/Green-Software-Foundation/if/blob/main/examples/manifests/group-by.yml

Also load any credentials via environment variables rather than embedded in the manifest file ;) see here for an example https://github.com/Green-Software-Foundation/if-unofficial-plugins/blob/8ddefffd4af0fc89f5763e31d46a887aeda314a6/src/lib/watt-time/watt-time-api.ts#L30C1-L33C11

tianbiao commented 2 months ago

Great - this would definitely be a useful plugin! Have you looked into what usage metrics are available from GCP APIs?

Hi @jmcook1186 there are a lot of usage metrics are available, I am going to start with Google compute engine usage metrics https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute