Measuring & Visualising the environmental impact of LLMs using the Impact Framework

[JadeMCap]

Prize category

Best Contribution

Overview

Introduction

Our project aims to tell a compelling story of the environmental impact of LLMs (Large Language Models). We feel this is such an important topic to look at because of the increased demand for, and interest in, LLMs across an assortment of different industries. We are also curious about how this rising demand fits in with company's environmental responsibilities and pledges.

The Technical

• We plan to use existing IF plug-ins to measure the carbon and energy impact of various LLMs.
• We plan to test for up to one hour.
• We plan to complete our analysis on multiple use cases: RAG, classification, one shot learning, email generation, tweet generation, grammar correction, generation product details, paraphrasing and summary and information extraction.

The Visual

• We then plan to create dynamic visuals within our github page to show the carbon and energy impact of the LLMs we test.
• The github page will include an animated chart - 2/3 mins run to simulate a real 1hr LLM service run.
• We plan to use the D3 library in Javascript to make this.

Questions to be answered

Can we submit our project for more than one award category? We feel it fits in with a few of the categories quite well. :octocat:

Have you got a project team yet?

Yes and we aren't recruiting

Project team

@JadeMCap @BaldelliMichele @chivchila @wispyplant @arun-apad @ogbk @blazarblast

Terms of Participation

• [X] I agree to the hackathon Rules & Terms and Code of Conduct

Submission

Summary

A brief overview of your project 100 words max

Our project has two stages:

Stage 1 – using existing IF plugins to measure the carbon and energy impact of running LLMs. We plan to use LLMs carrying out various use cases running for one hour where possible.

Stage 2: creating a website to show the carbon and energy use associated with the LLMs we test. This will allow non-technical audiences to consume the information output by the plugins and view six different metrics. The aim of this exercise is to allow users to visually see the energy/carbon use of LLMs and make decisions based on the results. Hopefully educating users or the amount of energy needed to power the tech.

Problem

Describe the problems the solution addresses 200 words max

• This project has been inspired by the increasingly frequent discussions about the environmental impact of LLMs.
• LLMs are exploding in use across multiple different sectors and are set to become even more prevalent over the next few years.
• This issue was again brought into the mainstream eye in the past few months following Sam Altman's (CEO of Open AI) comments stating nuclear fusion will be needed to power the massive energy needs of AI models.
• There is a conflict between company's increased desire to incorporate the benefits of AI (specifically LLMs) into their operations and their environmental responsibilities and pledges.
• There is also an issue of knowledge – not all user may be aware of the considerable energy and carbon consequences of LLMs.
• Our team wanted to explore this problem and propose a potential solution by investigating both the carbon and energy impact of LLMs and creating accompanying visuals.
• IF is designed on a low level infrastructure, so may not be fully assessable to non-technical users. By creating a web page with visuals we hope to open up the IF technology to a wider audience – which is essential if LLMs come into wider use.

Application

Describe what the solution actually does 200 words max

• We aim to measure the energy consumption and carbon footprint for a containerized LLM service, to which a different prompt is sequentially given for a predefined duration of time or a predefined workload and a description of the topic related to the submitted question.
• The solution consists in a containerized stack including cAdvisor (open source Docker container monitoring framework), Prometheus and the IF plugin / Python scripts for both extracting data from the Prometheus TSDB and preparing the data feed.
• The stack has been deployed to a GCP VM instance and is operated by executing a docker compose command, which kicks-off the simulation of the LLM service. It is the LLM service itself invoking the monitoring / calculation stack at the end of the simulation, which in turn delivers the final outputs.
• We have decided to complement its results with a UI platform, in order to effectively communicate its results to a wider range of audiences.
• We have created a web page using javascript (D3 library predominantly) that displays six time series charts. The output of the IF plugin can be added to a central .json file which then links to the webpage.

Prize category

Specify which prize category you are entering

Best Contribution

Judging Criteria

Explain how what you built meets the judging criteria for your prize category e.g. For Beyond Carbon - "Overall Impact", "Educational Value", "Synthesizing" Max 200 words

• Overall Impact: We believe the development of accompanying visuals open up the IF to a wider range of audience, thus widening the scope of using this tool to educate large groups on the carbon and energy consumption of LLMs. In order for the project to reach its full potential more work needs to be done on automating the process of passing the IF output to the website.
• Innovation & Creativity: Our project has a unique two stage approach which combines an interesting and topical use case with a practical and scalable enhancement to the IF (I.e., an accompanying website).
• Alignment: IF philosophy was a key consideration for us during the project.
  • We aimed to make our pipeline and our visuals flexible, verifiable and transparent so others can use our work.
  • The tool is very modular and scalable.
  • We aimed for our project to be educational and neutral, without providing any comment on what a 'good; output is.
• User Experience: The website we have created is simple to understand and has a clear, accessible design. We also aimed to create a simple website that has the flexibility to be customised by future users as need, in line with IF philosophy.

Video

A link to your video submission on YouTube

https://www.youtube.com/watch?v=tla1wd2QG-E&feature=youtu.be

Artefacts

Link to the code or content

https://github.com/ogbk/capgemini_ch2024

Usage

Link to usage instructions if applicable

N/A - on github page (link above)

Process

Describe how you developed the solution Max 150 words

• We organised ourselves into various sub teams - a web development team, AI focused team and storytelling team.
• We decided upon the tools and languages we planned to use in our project, after considering our key aims of educating our audience and combining the topics of LLMs and the Impact Framework.
• We carried out research on the IF, D3 library and LLM use cases.
• The IF was containerized with Python runtime providing pre-processing and post-processing capabilities (logical rendering of the IF manifest file and processing of its outputs).
• The output was saved to a .json file.
• We then created a website using Javascript, which we linked to our project Github Repo. The website connects to the data input into the .json file, which has to be manually changed for new inputs.
• The final stage was to carry out testing, first of dummy data and then real LLM outputs.

Inspiration

Tell us what inspired you to develop the solution Max 150 words

• There has been a significant increase in interest in LLMs by businesses across various sectors, which is predicted to continue to grow.
• This is happening at the same time as businesses are experiencing an increase in sustainability reporting requirements (e.g., the EU Corporate Sustainability Reporting Directive)
• There is rising concern about the vast amounts of energy that will be required to power the rise of LLMs (and AI more generally), which Sam Altman (ChatGPT creator) referenced in his recent speech as Davos.
• We have identified a potential gap between these two trends which could be filled with a tool like the Impact Framework.
• Our project is inspired by the need to balance the benefits of the continued rise and development of AI (particularly LLMs) with the environmental impact of the massive amount of computing power needed.
• We hope our project will inform and educate a wide audience, technical and non-techy.

Challenges

Share the challenges you ran into

• One challenge we had from early on in the project was the decision on how to best visualise the data. We had initially planned to use a dashboard tool such as Power BI, but later decided that this may provide barriers for future upscaling. The use of an external tool like this may also not be the most appropriate for a project that has a focus on green software development.
• The second challenge we faced was deciding which LLM use cases to test. We were mindful of incorporating different use cases and levels of complexity whilst also being mindful of time and relevance.
• The biggest challenge which we have not yet solved, is automating the process of moving the data output by the plugin into the website. At present we have to manually change the data in the output.json file. Ideally we would like this process to be automatic, perhaps by using a server to store the output and pass it to the website. Automating is the next challenge to be tackled if this project is to be used my non-technical audiences in the future.
• Finally, on a more human level, it was often challenging to balance the work required on the project with our other responsibilities. We overcame this by having regular update meetings and open communication, following a scrum style approach.

Accomplishments

Share what you are most proud of Max 150 words

• Creating a project in just three weeks using a completely new software with a team of people from varied backgrounds.
• Creating a website that is actually useful and has the potential to be developed into something with a real use. One of our main stumbling blocks was focusing our minds to create an MVP, as we were all full of ideas on how to take this project forwards! So we are proud to be able to submit something that is coherent and interesting.
• Submitting a project that may be the starting point for a future project that could have a massive impact on educating people on the environmental impact of LLMs. Potentially developing a tool that will help promote environmental consciousness.

Learnings

Share what you learned while hacking Max 150 words

• We gained significant knowledge of the Impact Framework, when compared to our starting point of zero knowledge. Part of the team learned more about the technical implementation and how to use the IF, whilst others became familiar with the IF's core philosophy and concepts.
• We learned about some of the of the challenges that can arise when working with more than one software type/tool/language and potential ways to overcome these (mainly by being adaptable and flexible when working). At various points during the pipeline we had challenges with integration and knowledge gaps, we managed to overcome the majority of these during the timeline and have plans to work on the remaining pressure points after the hackathon.
• We cemented our understanding in the part technology has to play in moving towards a more environmentally conscious planet.
• We also learned how fast three weeks can go by!

What's next?

How will your solution contribute long term to the Impact Framework eco-system Max 200 words

• The whole stack could represent the backend for a website or web portal, available to users for insights, also based on a GitHub repo, able to receive updated data to be rendered / further analysed.
• It could also be used to develop an Environmental Impact Calculator as a Service, part of a wider service providing ESG reporting.
• The solution is scalable and modular, which allows to de-couple the observability/computing stack (backend) from the measured software application (e.g.: the Docker compose configuration file could be logically rendered to include more containerized apps to be monitored by the same stack or it could be further reduced to just deploying the back end).
• Since the containerised IF plugin requires just 256MB, it makes this solution also suitable for edge devices (e.g.: Raspberry Pi, industrial micro-controllers, etc.), this is something to look at going forwards.

[russelltrow]

@JadeMCap in answer to your question on submission - You are only able to submit for one category, if your project spans multiple please chose the one where the judging criteria best demonstrate the strength of your submission.

[JadeMCap]

Thanks @russelltrow we have updated the award category to best contribution instead then. Thanks!