search

green-coding-solutions / eco-ci-energy-estimation

Eco CI Energy estimation for Github Actions Runner VMs
MIT License
48 stars 10 forks source link

readme

Eco-CI

Eco-CI is a project aimed at estimating energy consumption in continuous integration (CI) environments. It provides functionality to calculate the energy consumption of CI jobs based on the power consumption characteristics of the underlying hardware.

Requirements

Following packages are expected:

Usage

Eco-CI supports both GitHub and GitLab as CI platforms. When you integrate it into your pipeline, you must call the start-measurement script to begin collecting power consumption data, then call the get-measurement script each time you wish to make a spot measurement. When you call get-measurment, you can also assign a label to it to more easily identify the measurement. At the end, call the display-results to see all the measurement results, overall total usage, and export the data.

Follow the instructions below to integrate Eco-CI into your CI pipeline.

GitHub:

To use Eco-CI in your GitHub workflow, call it with the relevant task name (start-measurement, get-measurement, or display-results). Here is a sample workflow that runs some python tests with eco-ci integrated.

name: Daily Tests with Energy Measurement
run-name: Scheduled - DEV Branch
on:
  schedule:
    - cron: '0 0 * * *'

permissions:
  read-all

jobs:
  run-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Start Measurement
        uses: green-coding-solutions/eco-ci-energy-estimation@v3 # use hash or @vX here (See note below)
        with:
          task: start-measurement
        # continue-on-error: true # recommended setting for production. See notes below.

      - name: 'Checkout repository'
        uses: actions/checkout@v3
        with:
          ref: 'dev'
          submodules: 'true'

      - name: Checkout Repo Measurement
        uses: green-coding-solutions/eco-ci-energy-estimation@v3 # use hash or @vX here (See note below)
        with:
          task: get-measurement
          label: 'repo checkout'
        # continue-on-error: true # recommended setting for production. See notes below.

      - name: setup python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
          cache: 'pip'

      - name: pip install
        shell: bash
        run: |
          pip install -r requirements.txt

      - name: Setup Python Measurment
        uses: green-coding-solutions/eco-ci-energy-estimation@v3 # use hash or @vX here (See note below)
        with:
          task: get-measurement
          label: 'python setup'
        # continue-on-error: true # recommended setting for production. See notes below.

      - name: Run Tests
        shell: bash
        run: |
          pytest

      - name: Tests measurement
        uses: green-coding-solutions/eco-ci-energy-estimation@v3 # use hash or @vX here (See note below)
        with:
          task: get-measurement
          label: 'pytest'
        # continue-on-error: true # recommended setting for production. See notes below.

      - name: Show Energy Results
        uses: green-coding-solutions/eco-ci-energy-estimation@v3 # use hash or @vX here (See note below)
        with:
          task: display-results
        # continue-on-error: true # recommended setting for production. See notes below.

GitHub Action Mandatory and Optional Variables:

Electricity Maps Token

We use https://app.electricitymaps.com/ to get the grid intensity for a given location. This service currently works without specifying a token but we recommend to still get one under https://api-portal.electricitymaps.com/

You will need to set this token as a secret ELECTRICITY_MAPS_TOKEN. See the documentation how to do this https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions

You will also need to set it in your workflow files where you call display-results and get-measurement:

  - name: Eco CI Energy Estimation
    uses: ./
    env:
      ELECTRICITY_MAPS_TOKEN: ${{ secrets.ELECTRICITY_MAPS_TOKEN }}
    with:
      task: display-results
      pr-comment: true

Continuing on Errors

Once you have initially set up Eco-CI and have given it a test spin we recommend running our action with continue-on-error:true, as energy and CO2 metrics is not critical to the success of your workflow, but rather a nice feature to have.

      - name: Eco CI Energy Estimation
        uses: green-coding-solutions/eco-ci-energy-estimation@v3
        with:
          task: final-measurement
        continue-on-error: true

Consuming the Measurements as JSON

For both tasks get-measurement and display-results the lap measurements and total measurement can be consumed in JSON format. You can use the outputs data-lap-json or data-total-json respectively.

You must set json-output to true in GitHub or export ECO_CI_JSON_OUTPUT="true" for it to be active.

Here is an example demonstrating how this can be achieved:

      # ...
      - name: 'Checkout repository'
        uses: actions/checkout@v3
        with:
          ref: 'dev'
          submodules: 'true'

      - name: Checkout Repo Measurment
        uses: green-coding-solutions/eco-ci-energy-estimation@v3
        id: checkout-step
        with:
          task: get-measurement
          label: 'repo checkout'

      - name: Print checkout data
        run: |
          echo "total json: ${{ steps.checkout-step.outputs.data-lap-json }}"

      - name: Show Energy Results
        uses: green-coding-solutions/eco-ci-energy-estimation@v3
        id: total-measurement-step
        with:
          task: display-results

      - name: Print total data
        run: |
          echo "total json: ${{ steps.total-measurement-step.outputs.data-total-json }}"

Note that the steps you want to consume the measurements of need to have an id so that you can access the corresponding data from their outputs.

Note on private repos

If you are running in a private repo, you must give your job actions read permissions for the GITHUB_TOKEN. This is because we make an api call to get your workflow_id which uses your $GITHUB_TOKEN, and it needs the correct permissions to do so:

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      actions: read
    steps:
      - name: Eco CI - Start Measurement
        uses: green-coding-solutions/eco-ci-energy-estimation@v3
        with:
          task: start-measurement

Support for dedicated runners / non-standard machines

This plugin is primarily designed for the GitHub Shared Runners and comes with their energy values already pre-calculated.

All the values for supported machines are found in the power-data folder.

The heavy work to get this values is done by Cloud Energy (See below for details).

If you want to support a custom machine you need to create one of these files and load it into Eco-CI.

Here is an exemplary command to create the power data for the basic 4 CPU GitHub Shared Runner (at the time of writing 13. June 2024).

python3 xgb.py --tdp 280 --cpu-threads 128 --cpu-cores=64 --cpu-make "amd" --release-year=2021 --ram 512 --cpu-freq=2450 --cpu-chips=1 --vhost-ratio=0.03125 --dump-hashmap > github_EPYC_7763_4_CPU_shared.sh

The following would be the command for Gitlab Shared Runners (at the time of writing 13. June 2024)

python3 xgb.py --tdp 240 --cpu-threads 128 --cpu-cores=64 --cpu-make "amd" --release-year=2021 --ram 512 --cpu-freq=2250 --cpu-chips=1 --vhost-ratio=0.015625 --dump-hashmap > gitlab_EPYC_7B12_saas-linux-small-amd64.txt

Gitlab uses an AMD EPYC 7B12 according to our findings

You can see how the machine specs must be supplied to Cloud Energy and also, since the runners are shared, you need to supply the splitting ratio that is used.

Since GitHub for instance uses an AMD EPYC 7763, which only comes with 64 cores and 128 threads, and gives you 4 CPUs the assumption is that the splitting factor is 4/128 = 0.03125.

An uncertainty is if Hyper-Threading / SMT is turned on or off, but we believe it is reasonable to assume that for Shared runners they will turn it on as it generally increases throughput and performance in shared environments.

If you have trouble finding out the splitting factor for your system: Open an issue! We are happy to help!!

Once you have the file ready we are happy to merge it in through a PR! In future versions we also plan to include a loading mechanism, where you can just ingest a file from your repository without having to upstream it with us. But since this is a community open source plugin upstream is preferred, right :)

GitLab:

To use Eco-CI in your GitLab pipeline, you must first include a reference to the eco-ci-gitlab.yml file as such:

include:
  remote: 'https://raw.githubusercontent.com/green-coding-solutions/eco-ci-energy-estimation/main/eco-ci-gitlab.yml'

and you call the various scripts in your pipeline with call like this:

- !reference [.<function-name>, script]

where function name is one of the following: start_measurement - begin the measurment get_measurement - make a spot measurment here. If you wish to label the measurement, you need to set the ECO_CI_LABEL environment variable right before this call. display_results - will print all the measurement values to the jobs-output and prepare the artifacts, which must be exported in the normal GitLab way.

By default, we send data to our API, which will allow us to present you with a badge, and a front-end display to review your results. The data we send are: the energy value and duration of measurement; cpu model; repository name/branch/workflow_id/run_id; commit_hash; source (GitHub or GitLab). We use this data to display in our green-metrics-tool front-end here: https://metrics.green-coding.io/ci-index.html

If you do not wish to send us data, you can set this global variable in your pipeline:

variables:
  ECO_CI_SEND_DATA: "false"

Then, for each job you need to export the artifacts. We currently export the pipeline data as a regular artifact, as well as make use of GitLab's Metric Report artifact (which we output to the default metrics.txt):

artifacts:
    paths:
      - eco-ci-output.txt
      - eco-ci-total-data.json
    reports:
      metrics: metrics.txt

Here is a sample .gitlab-ci.yml example file to illustrate:

image: ubuntu:22.04
include:
  remote: 'https://raw.githubusercontent.com/green-coding-solutions/eco-ci-energy-estimation/main/eco-ci-gitlab.yml'

stages:
  - test

test-job:
  stage: test
  script:
    - !reference [.start_measurement, script]

    - sleep 10s # Your main pipeline logic here
    - export ECO_CI_LABEL="measurement 1"
    - !reference [.get_measurement, script]

    - sleep 3s # more of your pipeline logic here
    - export ECO_CI_LABEL="measurement 2"
    - !reference [.get_measurement, script]

    - !reference [.display_results, script]

  artifacts:
    paths:
      - eco-ci-output.txt
    reports:
      metrics: metrics.txt

How does it work?

Limitations / Compatibility

See also our work on analysing fixed frequency in Cloud Providers and CI/CD

Note on the integration / Auto-Updates

Testing

For local testing you can just run in the docker container of your choice, directly from the root of the repository:

docker run --rm -it -v ./:/tmp/data:ro invent-registry.kde.org/sysadmin/ci-images/suse-qt67:latest bash /tmp/data/local_ci.example.sh