Integrate Impact Framework into load & performance tests

joerg-drescher commented 3 months ago

Prize category

Best Content

Overview

Why? It would be great to enable development teams to directly integrate Impact Framework into their performance tests, because:

Performance and load tests should in general play one important role in measuring how green a peace of software is.
Enabling the teams to not only measure response times and throughput in performance test but also directly relate the software under test to it's carbon footprint could become a motivator to optimize for minimizing it.
Also the boundary problem of measuring the environmental impact of software should be more easy to handle in the context of performance tests, as per-se teams typically pay well attention to have well defined and repeatable conditions during performance testing.
Last- not least: In case the load&performance tests are run regularly, improvements regarding the CO2 footprint can be directly compared over time. Also deployments which would drastically increase the environmental impact can be more easily discovered.

How?

Develop two plugins for the data exchange between Performance test tools (i.e. JMeter, Gatling or K6) and the Impact Framework: One "Import plugin" for transferring the conditions and parameters of the load tests scripts to IF to clearly describe the boundaries a second "Export Plugin" to transfer results of impact calculations of the IF back to the performance tests tools to enhance the results reports.
Develop a template/example manifest combining these import and export plugins with other IF plugins to monitoring and impact calculation plugins (we could start with the SCI plugin).

Questions to be answered

Do we already have monitoring Plugins in the IF? The scope would for sure become to big in case we would need to also develop those. I only see the proposal of a Prometheus plugin in the list of proposed projects for this hackathon.
I have not secured my team yet (speaking with 2-3 former colleagues) and myself I'm non-technical, so we might need to recruit contributors.
Can this project proposal be brought-up and briefly be discussed in the next Live Prep and Q&A session coming monday (March 11th) to get an idea if it's worth following (and still try to get a team)? - I Hope you spot it before....

Have you got a project team yet?

Yes

Project team

@yannlv ; @stephane-batteux

Terms of Participation

[X] I agree to the hackathon Rules & Terms and Code of Conduct

Project Submission

Summary

We explored how we could utilize the Impact Framework to obtain carbon emissions of applications while doing load and performance tests.

We demonstrate the concept on how to expand a K6 Performance test script to transfer the test parameters and results as input parameters for the GSF Impact Framework and to calculate the operational carbon emitted during the performance test run. The actual demo scope shows it for the example of the e-net plugin calculating the network energy.

Problem

User Story: As a load tester I want to integrate my performance tests with GSF Impact Framework, so that I can get carbon emissions of the application as part of my performance test results.

Load and performance tests are executed under well defined conditions and with repeatable load inputs. Therefor they are an ideal use case for the utilization of the IF as they inherently solve the border challenge. Also this provides an easy approach to directly measure improvements over time regarding carbon efficiency when load and performance tests are repeated on a regular base.

Application

We created an example of an integration of IF into performance tests using K6, which is one of the most common used performance test tools. To achieve the goal we expanded the performance script to generate a IF Manifest and invoke it through the command-line as last step of the test process. The performance test parameters and data volumes get transferred into the Manifest as input parameters for the carbon calculation.

Prize category

Best Content

Judging Criteria

Overall Impact: The easy utilization of IF in the context of load & performance test generates a great potential regarding the actual usage of IF.

Load and performance tests are already a common practice across the industry.
Directly related to the resources-utilization.
Load and performance tests are executed under very well-defined conditions, giving a well understanding of the border of the application observed.
Making the carbon measurement repeatable.
The conceptual approach should be easily transferable to other performance test tools.

Clarity: The solution is simply a short expansion of the performance tests script. Anyone already familiar with the K6 should be able to easily follow the example.

Innovation: This project brings IF right into the best spot of the software development lifecycle to learn regarding the carbon emissions and motivate to improve!

Video

https://youtu.be/5-LSVtmG1vA

Artefacts

Code: https://github.com/yannlv/k6-perfQA4impact Presentation: https://www.dropbox.com/scl/fo/sn7ubp5y3h9nivyyi3gnk/h?rlkey=k5kc9gb3xtrj0ts0ms32t81gh&dl=0

Inspiration

Load and Performance testing appeared as the best natural fit to approach carbon measurement of running software applications. Therefor we wanted to aim at integrating IF into this task.

Challenges

There are no monitoring plugins available yet. So the approach for now is based on mock data regarding CPU and memory. The integration with real monitoring data in addition to the performance test outputs will need to come in a next step but should be easy after those become available.

Accomplishments

We lay out an easy approach to expand the K6 script to be able to transfer the parameters and outputs of the performance tests to the IF and get the operational carbon emissions in return. The demo shows the principle by invoking the e-net plugin, using test parameters and data volume results from the performance test.

Learnings

We first wanted to approach the problem by creating an importer plugin but quickly realized that it could be more suitable to be able to invoke the execution of IF by the performance test itself. So, we came to the idea to seek a way to generate the manifest and invoke it through command-line at the end of the test run.

What's next?

Expand the script to deliver the operational carbon. Integrate coming monitoring plugins (i.e. Prometheus Importer) into the solution so the real CPU and Memory usage during the test run will get considered. Transfer the solution to other performance test tools (I.e. JMeter or Gatling).

nithyeswari commented 3 months ago

Are you still looking for an extra member?

iamkennis commented 3 months ago

Are you looking for someone to join the team

Sophietn commented 3 months ago

Hi @joerg-drescher I see you have the recruiting label on your issue. Are you looking for additional team participants? Would you like to attend our Launch event on the 18th March 2.30pm GMT, to discuss your project idea and attract team members?

joerg-drescher commented 3 months ago

Hi Amodu and Nithyeswari: In principle I'm still looking. Do you have any experience with perfprmance test tools (i.e. Gatling, JMeter, K6)?

jawache commented 3 months ago

Hi @joerg-drescher great chatting on monday! The way I would perhaps approach this is to create an importer/runner plugin which ran the tests, captured observations and then returned those to the IF for processing by other plugins in the pipeline.

So a plugin perhaps configured like so

    script-monitor:
      path: 'your-plugin-path'
      method: TestRunner
      global-config:
        command: a shell script to execute

The plugin runs the shell script defined in command, but as well as running the script it perhaps also keeps a close eye and monitors the cpu utilization while running the script and then returns something like so

  outputs:
    - timestamp: '2024-03-05T00:00:00.000Z'
      duration: 360
      cpu/utilization: 54
    - timestamp: '2024-03-05T00:06:00.000Z'
      duration: 360
      cpu/utilization: 22

And then passes that onto other plugins in the pipeline which then compute the environmental impacts of the script.

If you wanted to publish this data back to some central test/perf db I would write a script that takes as input the computed yaml file, parses it and then publishes the data wherever you want.

You could start with a phase 1 where it just monitors the machine the script is run on, but if you're script applies load to a cloud environment to multiple processes/machines you could in a second phase expand it out to extract cpu/utilization from all those disparate machines and pull it back but again, just output them all as observations which other plugins in the pipeline would process.

That kind of approach would get you somewhere quick, but then also give you options to expand later on (baby steps!).

iamkennis commented 3 months ago

Hi Amodu and Nithyeswari: In principle I'm still looking. Do you have any experience with perfprmance test tools (i.e. Gatling, JMeter, K6)?

No, I dont

nithyeswari commented 3 months ago

I do have experience in JMeter..Loadrunner.

On Wed, 13 Mar 2024, 12:27 Jörg Drescher, @.***> wrote:

Hi Amodu and Nithyeswari: In principle I'm still looking. Do you have any experience with perfprmance test tools (i.e. Gatling, JMeter, K6)?

— Reply to this email directly, view it on GitHub https://github.com/Green-Software-Foundation/hack/issues/88#issuecomment-1994272774, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACX7GBIJXYLAXO2VBBAVG4LYYBA2LAVCNFSM6AAAAABEMU7WCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJUGI3TENZXGQ . You are receiving this because you commented.Message ID: @.***>

joerg-drescher commented 3 months ago

We're not recruiting anymore.

Green-Software-Foundation / hack