search

thegreenwebfoundation / grid-intensity-go

A tool written in go to help you factor carbon intensity into decisions about where and when to run computing jobs.
Apache License 2.0
61 stars 7 forks source link

Figure out examples of reference architecture for supporting the Real Time Carbon Footprint Standard #75

Open mrchrisadams opened 1 year ago

mrchrisadams commented 1 year ago

Hey @rossf7 - I have an idea how you might implement reporting based on this Real Time Carbon Footprint Standard, but I figured it would make sense to start a convo in the open about what a reference architecture might look like.

Here's the proposal: https://github.com/orgs/Green-Software-Foundation/discussions/34

And here's the original writeup https://medium.com/@adrianco/proposal-for-a-realtime-carbon-footprint-standard-60b71c269948

And https://github.com/adrianco/slides/blob/master/Cloud%20DevSusOps%20London.pdf The deck is really worth a look, and the green web fdn might be in a particularly interesting place to surface some of this info, given we ask providers for information about how they are accounting for their emissions and they often tell us about how which RECs they're purchasing and so on.

The PR FAQ is worth reading too https://github.com/Green-Software-Foundation/standards-wg/issues/67

I'll add some notes here

How I think we could do this with code now:

You can use the Green Web Exporter for Grafana to provide CO2 figure for every hour - it’s got integrations with existing providers of data, and we’re friendly with them: https://github.com/thegreenwebfoundation/grid-intensity-go/

We know about Scaphandre, which gives the very precise, per process figures at very high temporal resolution for power, along with the ability to tag various processes. https://github.com/hubblo-org/scaphandre/blob/main/docs_src/explanations/how-scaph-computes-per-process-power-consumption.md

I think Kepler essentially does the same thing, providing high resolution figures, but at the container level instead, and via ebpf. I'm less familiar with the internals, and how tightly it is tied to k8s and containers: https://github.com/sustainable-computing-io/kepler

Any chance we can try sketching this out on a call?

mrchrisadams commented 1 year ago

For future reference:

This paper is the first I've seen that tried to address the assumptions made by lots of energy tracking tools:

https://hongyu.nl/papers/2023_hotcarbon_energat.pdf

Check it - the diagram in the paper specifically refers to disaggregated compute and storage - I haven't seen any tooling explicitly refer to how this assumption impacts readings:

Screenshot 2023-07-20 at 12 48 28

Speaking of which, the numbers from this new tool seem significantly higher than Scaph and other tools. I wonder how Kepler compares? Screenshot 2023-07-20 at 12 48 37

Here's the paper: https://hongyu.nl/papers/2023_hotcarbon_energat.pdf

and here's the video https://mediaspace.ucsd.edu/media/HotCarbon%E2%80%9923%3A%20EnergAt%3A%20Fine-Grained%20Energy%20Attribution%20for%20Multi-Tenancy%20(He%20et%20al.)/1_ac1bpnvt/307441832