[Action] Tracking the energy consumption of GitHub Actions Pipelines

[by-d-sign]

Pipelines are not only a great tool for automating tedious tasks, they are also probably a mayor energy consumer. Optimizing them could be worthwhile, however their actual energy demand is currently neither visible nor well researched.

Eco-CI is an approach to measure/estimate the energy consumption of a GitHub Actions Pipeline. This could be a great tool to raise awareness and a first step towards optimization.

[ArneTR]

Thanks for mentioning our tooling @by-d-sign !

We are very happy to make a PR for a project of your choice. We wil integrate it on a fork and then generate some data so you have an idea how it looks like.

Which project shall it be? @nikimanoledaki @by-d-sign :)

[leonardpahlke]

Sounds like a great subject for a blog post @by-d-sign @ArneTR 😉 https://github.com/cncf/tag-env-sustainability/issues/new?assignees=&labels=blog&projects=&template=blog.md&title=%5BBLOG%5D+%3Cblog-title%3E

[leonardpahlke]

cc @savitharaghunathan

[AntonioDiTuri]

As we agreed during the WG regular meeting this is out of scope for the current milestone, thanks for adding the issue. We will circle back with a proposed action time after the KubeCon when we will have time to organize and prioritize issue for the next milestone. Info about the milestones are currently in the readme

[ArneTR]

I made a demo of how this would look for one of the workflows from Falco. I picked the only one that had a cron, as this shows the most simple integration:

• Workflow chosen: https://github.com/falcosecurity/falco/actions/workflows/scorecard.yaml
• Draft PR where the demo is integrated to show diff: https://github.com/falcosecurity/falco/pull/3121
• Generated data by Eco-CI: https://metrics.green-coding.io/ci.html?repo=green-coding-solutions/falco&branch=master&workflow=87453986

Additonal: @by-d-sign Eco-CI can also handle matrix/parallel Github Actions out of the box. Since you brought this up as a question in one of the last community meetings :)

[ArneTR]

Hey folks,

I hope you had a great KubeCon.

We have recently added some features to our Eco-CI tooling. Now it shows also the location of the running machine which then enables to include grid intensity as well as total CO2e emitted.

You can see it in this sample PR that we have made on a fork from the Falco repository.

Aggregation and timeline data is provided here: https://metrics.green-coding.io/ci.html?repo=green-coding-solutions/falco&branch=2/merge&workflow=87453986

Hope this ping is interesting for you :)

@ribalba: Looping you in

[leonardpahlke]

@nikimanoledaki @AntonioDiTuri ^

[ArneTR]

Hey folks,

after I month I thought I give this a gentle ping. Any update if you fancy to integrate Eco-CI into any projects?

We are super happy to help and provide a PR. Integration effort should be minimal.

We recently also approached other open source projects and can provide ready tested PRs. Example : https://github.com/NMF-earth/nmf-app/pull/393

While doing so very often also some direct optimizations come into view. Example where we did this with NMF: https://github.com/NMF-earth/nmf-app/pull/395

[nikimanoledaki]

Hi @ArneTR! A PR is very much welcome.

If you get stuck:

• Let one of us know if there's anything needed to help -> @cncf-tags/tag-env-wg-green-reviews-leads
• Feel free to share this on the CNCF Slack channel to inform or collaborate with the other WG contributors.
• You are welcome to join one of the meetings to discuss this further!

We are working on some Github Action pipelines as part of two proposals being written right now. One is open for a review, for an idea of how we'll be using GA pipelines: https://github.com/cncf-tags/green-reviews-tooling/pull/88 - also feel free to review if you'd like!

Looking forward to reviewing your PR 👍 😊

[ArneTR]

Hey @nikimanoledaki

what I would need is a point which repo you find most suitable. You have quite many in the CNCF :)

In the history of this Issue I already pointed to a sample integration for Falco (See here) Which was however a very small pipeline and just for demo purposes.

So a pointer would be nice for a repo:

• That you, maybe even personally, interact with a lot and thus the integration would be very visible to you and provide the most value for now
• A pipeline that runs at least once a day and at least 2 minutes. Longer is better as data becomes less noisy and more actionable

I will make a PR then and tag you there

[wbeckler]

Would you consider submitting this to https://github.com/opensearch-project/OpenSearch/ ? It's a beefy build that takes a long time and runs a lot.

[ArneTR]

Would you consider submitting this to https://github.com/opensearch-project/OpenSearch/ ? It's a beefy build that takes a long time and runs a lot.

Hey @wbeckler, sure! Which of the pipelines do you have in mind? Project has quite many apparently :)

[ArneTR]

Hey @nikimanoledaki @leonardpahlke @wbeckler @by-d-sign

Since I get no reply on my last message I thought I go ahead and just pick the biggest workflow and make a sample integration there.

Here is the PR: https://github.com/opensearch-project/OpenSearch/pull/14450

I also tagged you there.

The implementation features the newest Eco-CI with minimal overhead (https://github.com/green-coding-solutions/eco-ci-energy-estimation/releases/tag/v4.0-rc3) but I refrained from integrating stuff like PR-Comments as this would require workflow permissions changes and I wanted to keep it slim.

Happy for your feedback!