Green-Software-Foundation / sci-guide

Open Data project will provide all the necessary data sources that can be used as inputs for the SCI standard, for free.
https://sci-guide.greensoftware.foundation/
Other
6 stars 12 forks source link

[Investigative Submission]: Energy consumption resources from different Cloud providers #17

Open Willmish opened 2 years ago

Willmish commented 2 years ago

Contact Details

GitHub/Slack: Willmish

Data request?

Currently the Carbon CI Pipeline Tooling project is facing a blocker as it is missing energy consumption/power usage data for different kinds of resources used by Cloud providers and cannot properly estimate carbon emissions of Cloud infrastructures.

What we need is data power usage/energy consumption for different resource types (databases, VMs, app services etc) for different cloud providers:

Currently we are focusing on adding support for Azure infra-as-code in particular as a starting point - getting data for those resources would be a good step towards a ready-to-use project and first release.

Outline any further information requirements

As mentioned in the data request, Azure data and estimation is our current focus, but as we expand to other Cloud providers we will need their power usage data as well.

Code of Conduct

mrchrisadams commented 2 years ago

hi @Willmish

The best place I know for this information right now is currently the cloud carbon footprint codebase, but it's worth understanding the tradeoffs being made there.

To my knowledge, no cloud providers release energy usage figures on a per product basis - the closest you will find is Google Cloud in their own cloud carbon footprint calculator, where if you know the region you are interested in, and the average location based carbon intensity figures from Google, then you can work backwards to come up with some rough numbers.

A little more info on the data from Google

You can see the emission constants here used for Google. https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/packages/gcp/src/domain/GcpFootprintEstimationConstants.ts

And the rough mappings of those figures here to instance types made available: https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/packages/gcp/src/lib/MachineTypes.ts

However, it's worth bearing in mind that even then, the numbers in the GCP own calculator appear to be based more on spend, than energy usage.

This keynote, at the hot carbon conference is the only public information I have been able to find

https://www.youtube.com/watch?v=W7uTbxCxmPg

Hot carbon conference: https://hotcarbon.org/program/

As far as I am aware, this also does not include embodied energy in the servers themselves.

This is your best bet for numbers right now.

Looking at Azure

If you're looking for Azure figures then it's likely more difficult, as while Microsoft's carbon calculator does include the embodied energy in the servers, in each service, and they provider a greater breakdown in terms of the kinds of products, I don't think they release figures on the energy use without adjusting it using the GHG Protocol Scope 2, market-based adjustments. So, you only see the figure for each product line after the scope 2 adjustments have been made, obscuring the energy usage.

This makes it much more challenging to work backwards to get the energy usage - your best bet here again is to use the numbers in Cloud Carbon Footprint, which in turn are based on SPEC figures for specific processors, rather than actual numbers from Microsoft.

Looking at AWS

Once again, your best numbers in the public domain are on Cloud Carbon Footprint, and those are based on the figures in the cloud carbon coefficients like the other providers:

https://github.com/cloud-carbon-footprint/cloud-carbon-coefficients

And once again, I'm not aware of figures you can use beyond that, that are published by Amazon - the top level carbon footprint figures in their calculator do not provide a per-product breakdown, nor a region breakdown.

In addition, you only have the carbon figures, after the same GHG Protocol scope 2 adjustment has been applied - so the location based energy use figures are obscured here too.

Without the location based energy usage figures, it's very difficult to work out the figures for the SCI once again.

Getting the numbers you need

We'd need the providers to release these energy usage figures to make it possible to work out the SCI scores in all these cases.

Our best bet is to speak to someone at Azure to ask them to release this information, and if they say no, we fallback to the numbers in cloud carbon footprint - at least that way, the assumptions being made are in the public domain.

jawache commented 2 years ago

@Willmish do you have specific examples you are looking for? VMs are the most straightforward and the links @mrchrisadams provided as the best first step, but managed services are a lot more challenging.

@srini1978 has done the most work here trying to model/estimate numbers for Azure managed services. I think we are going to have to come up with some rough models for managed services, or maybe the cloud carbon footprint team are looking into this or have some thoughts on this? @camcash17?