[Use Case Article] - Simple and automated SCI measurements

[ArneTR]

Simple and automated SCI measurements

Executive Summary

The Green Metrics Tool (GMT) is an open-source software solution that provides the capabilities of automating the process of generating an SCI metric for a given piece of software.

The GMT is essentially a container native benchmarking tool that instruments a software according to a so called usage scenario. Which describes the usual interaction with the software in an computer executable fashion. While executing this scenario different measurement sources are polled like machine energy, cpu energy, network traffic etc. and then put into context with a unit of work.

When supplying the basic input variables to the SCI like (I, EL, RS, TE) and instrumenting the software so that it outputs the unit of work variable (R) to a standard output stream the tool automatically generates the SCI metric.

All of this instrumentation is fixated in a typical DevOps like standard infrastructure as code (for instance Docker compose files) and can be part of the git repository. Thus, businesses can seamlessly integrate carbon footprint assessments into their software development pipelines, fostering a culture of sustainability.

Description of problem

When talking about the energy efficiency of a software it is a common approach to look at the work done and also at the total energy cost and put these metrics in relation to one another. The SCI goes one step further and also adds the embodied carbon as well as the current carbon grid intensity. However so far the SCI has to be manually derived and there is no reference implementation in an open source software that showcases the process and also makes it reproducible. This makes the SCI less approachable and also harder to replicate and validate. There is also no way to easily track SCI over time with a fine resolution as the measurement can not be automatically made.

The solution / How the use case solves the problem

By using an automated benchmark approach the process of generating the SCI metric is not only easier, but also reproducible. When the to be benchmarked software also is open source it additionally allows for the possbilty to validate the metrics by the open source community.

As a free and open-source software (FOSS) the Green Metrics Tool presents a first approach to an implementation of automatically and reproducible generating the SCI metric.

The tool not only provides all the software to set up a measurement locally, but also a reference hardware where measurements can be run for free with specialized measurement hardware and software. This allows everyone to trace back claims and also generates better results as it is not feasible for an individual or small company to invest in a dedicated measurement cluster, when just starting with green software practices.

Starting with a git repository the software needs a simple usage scenario file which is extensively documented and also many example implementations are provided. Once the usage scenario file has been added the repo can be submitted the the GMT cluster and will automatically benchmarked. It is also possible to run the benchmark on every commit or in any other timely matter. This enables a detailed tracking of the software over time. Also changes on the software are reflected right away in the SCI score published.

Example case - Wagtail

Wagtail is a popular open-source CMS based on Django which comes with a detailed reference implementation: The Bakery.

In a clone of the repository we added the needed instrumentation files and defined the needed SCI variables.

To make it reproducible we used the machines that are freely usable in the Green Metrics Tool measurement cluster and their respective lifetime and embodied carbon values.

As the unit of work in Wagtail we defined R as the per page visited, since it is a CMS and one of it's main use cases is offering a framework that creates websites that are mainly visited by a browser.

The SCI value we are calculating here, as an example, is the cost webpage viewed. The SCI here is 0.02 gCO2e/page request

Example case link: https://metrics.green-coding.berlin/stats.html?id=6e4936e0-2a78-4f5b-afe2-1299ed37a964

Exemplary SCI values for Wagtail page visits

Example case - Nextcloud Talk

Nextcloud is another very prominent open source software that allows to set up a Google Workspaces like platform on ones own infrastructure.

Here we used the standard Nextcloud docker images and then initiated a Talk session betwen multiple users. The scenario is run with Chrome browsers running in headless mode.

Also, to make it reproducible, we used one of the machines that are freely usable in the Green Metrics Tool measurement cluster and their respective lifetime and embodied carbon values.

The SCI value we are calculating here as an example is the cost per Talk message. The SCI here is 0.15 gCO2e/Talk Message

Example case link: https://metrics.green-coding.berlin/stats.html?id=84645f34-2195-43e2-8c61-dcb3afe37120

Exemplary SCI values for Nextcloud Talk

Further example cases

We have started benchmarking various open source projects. With the aim to see how the SCI developes over time and how design decisions affect the benchmark. Also it illustrates how simple it is to add the the SCI to a piece of software. The idea to monitor various projects is called Energy ID

At the moment this includes:

• Wagtail
• Wordpress
• Django
• Nextcloud

However, we also have other example projects to highlight the broad applicability

• Algorithmic / AI workloads
• APIs
• Idle scenarios

It's crucial to understand that energy consumption isn't a static metric but a dynamic one. As we now have a timeline over the development of the software we can create badges that showcase how the software performs in relation to resource usage. Thus hopefully making the metric a first class citizen in deployment.

Looking at the SCI over time

Why should you use automation to generate your SCI metric?

In today's rapidly evolving tech landscape, the need for sustainable, energy-efficient solutions has never been more pressing.

The SCI provides an easily digestible metric to get a first glimpse at the energy and carbon profile of your application.

By levering open-source tools this democratizes access to this vital metric, allowing a wider community of developers, researchers, and institutions to leverage it.

The power of automation cannot be understated. With manual processes often proving cumbersome and error-prone, our software streamlines the process by automatically generating the SCI metric. This not only saves valuable time but also ensures precision and consistency.

[ursvill]

@seanmcilroy29 - Sean, this looks like a good use case to me. I would like to suggest it for an article. Who can check the technical side of it?

[navveenb]

@ArneTR , please add your contribution to the Awesome Green Software project.

[seanmcilroy29]

@ursvill - WG have agreed for this submission to be written up as an article. @Henry-WattTime will provide technical direction for the article where required

[ursvill]

@Henry-WattTime - When would be a good time for you to go through the case with me? I am guessing about 20min?

[ArneTR]

If I can be of any further help here to advance the progress on this please let me know. Happy to help.

[ursvill]

@ArneTR - I will work on it this week; will let you know if I have any questions.

[ursvill]

@ArneTR - Hallo Arne, shared the link to my draft. Please make sure I didn't misstate anything. Also feel free make any additional comments you deem important. Gruesse Urs

[ArneTR]

Hey @ursvill ,

thanks for the great overhaul of the article! I swapped one link to our cluster to not link to the "install page" but rather to the actual machines in the cluster, which I think is more informative.

I do not know if you want that in your GSF articles, but since the GSF wants more use cases and this is why we also created this free tool to encourage this, something at the end like "So if that peeked your interest than try out this tool and submit your use case to the GSF to show that you care about software carbon transparency".

I specifically not added it to the article as I did not know if you want the GSF articles rather to be informative or engaging. So just an idea.

[ursvill]

@Henry-WattTime - Henry, can I ask for your technical review of this article please: https://docs.google.com/document/d/1iPLjX0fB6BX1uRPnGwpLr-SaSiwCUBbm/

[ArneTR]

@ArneTR , please add your contribution to the Awesome Green Software project.

Hey @navveenb quick question on the awesome software project: I added the software there a while ago. However the last release here to main was in Nov 2022. Who is maintaining that project internally at GSF?

[ArneTR]

Yay, the article got released https://greensoftware.foundation/articles/use-case-automated-sci-measurements-with-green-metrics-tool

How is your worfklow. Shall openers close the issue?

I did not want to go ahead on this but rather ping, that this issue can be closed.

[Henry-WattTime]

It can be closed! The full text of the case study is also recorded here: https://github.com/Green-Software-Foundation/sci-guide/blob/dev/use-case-submissions/Automated_SCI_Measurements_with_Green_Metrics_Tool.md