SamYuan1990
commented
9 months ago just provides information here...
Prometheus importer plugin - Pulls Kepler metrics from prometheus server
you can ref here, I hope it helpful. https://github.com/sustainable-computing-io/kepler/blob/main/e2e/integration-test/e2e_metric_test.go#L290
We would require Kepler and Prometheus/Grafana to be installed on the Kubernetes cluster
https://github.com/sustainable-computing-io/local-dev-cluster may help you to set up a kind(k8s in docker) based cluster for development usage, meanwhile which kepler developer used to development kepler by design. https://github.com/sustainable-computing-io/kepler-action invokes this repo to set up k8s cluster in kepler's CI.
Sophietn
greenscale-nandesh
tmcclell
Type of project
Building a plug-in for Impact Framework
Overview
Context
Kepler is a CNCF project for measuring energy consumption of bare metal Kubernetes clusters using RAPL. A Kepler model for Impact Framework would query Prometheus for Kepler metrics and allow calculating the SCI score for applications deployed to Kubernetes clusters when Kepler is deployed.
Idea/Problem statement
We are thinking of building a pipeline which has following steps
Scope
For hackathon, it would be limited to
Questions to be answered
Have you got a project team yet?
Yes and we are still open to extras members
Project team
@greenscale-nandesh @rossf7 @SFulpius @tmcclell
Terms of Participation
Project Submission
Summary
The Kepler Importer plugin is an essential tool for Kubernetes users who are serious about understanding the environmental impact of their workloads. With this powerful plugin, users can achieve a deeper insight into how their workloads affect the environment, and make informed decisions to reduce their impact.
Problem
Kepler (Kubernetes-based Efficient Power Level Exporter) is an impressive tool that leverages eBPF to probe performance counters and system stats. Using advanced ML models, it accurately estimates energy consumption based on these stats and exports them as Prometheus metrics. However, it currently lacks support for assessing the environmental impact of K8s workloads.
Application
The Kepler Importer plugin is a powerful tool that enables easy and efficient importing of Kepler metrics using Impact Framework. By leveraging this plugin, users can effortlessly calculate the SCI score using official plugins- a crucial metric for evaluating the environment impact of applications deployed to Kubernetes clusters. With its advanced features and user-friendly interface, the Kepler Importer plugin is the perfect solution for businesses looking to optimize their Kubernetes deployments.
Prize category
📦 Best Plugin
Judging Criteria
Overall Impact This plugin could increase the practice of measuring energy consumption in Kubernetes clusters to enable reducing their footprint. It could help increase the adoption of both Impact Framework and Kepler. This is important because in 2022 a report stated 70% of organizations had adopted Kubernetes.
Opportunity It opens the door to using Impact Framework and Kepler across both Kubernetes and the wider cloud native ecosystem. Kepler supports using both RAPL on bare metal and can use machine learning when deployed to the cloud and RAPL is not available. Our plugin supports both these modes.
Modular We have followed the Unix philosophy to keep a small scope and ensure the plugin just integrates with Prometheus. We also tested it with the other IF plugins to ensure they can use the energy value the plugin provides.
Video
https://www.youtube.com/watch?v=sn0zrhghrjE https://docs.google.com/presentation/d/1xXVTVBOXM1ZJXWYE67anPl9cWEwAis8-KZoJL4p-UM8/edit?usp=sharing
Artifacts
Plugin - https://github.com/greenscale-nandesh/if-unofficial-models/tree/main/src/lib/kepler-importer Manifest - https://github.com/greenscale-nandesh/if-unofficial-models/blob/main/kepler-pipeline-manifest.yaml
Usage
https://github.com/greenscale-nandesh/if-unofficial-models/blob/main/src/lib/kepler-importer/README.md
Process
We started by looking at the existing plugins and agreeing the design in a Miro board and then created GitHub issues. We then found a Prometheus SDK for TypeScript and used the WG Green Reviews cluster to test with Kepler deployed. We then created a manifest with a full SCI pipeline to integrate with the other plugins. During development we held twice a week team calls to sync with weekly calls prior to the start to plan.
Inspiration
Some of our teammates are part of CNCF Green Reviews working group. During the development of the monitoring pipeline they realized that having an Impact Framework plugin to generate SCI score for K8s workloads using Kepler would be useful for the working group and also the wider cloud native community.
Challenges
One challenge was to debug input and output between plugins. Initially we didn’t realize we needed to pass all inputs to the next stage in the pipeline. Another challenge was Kepler provides metrics in Joules which we needed to convert to kWh.
Accomplishments
Learnings
What's next?