Create CCF Cloud SCI IMP

jawache commented 1 year ago

Name

ccf.cloud.sci

Scope

This model returns the SCI score and component E, I and M values for cloud providers as supported by the CCF Package

References

Assumptions

N/A

Limitations

This is raw data, the modules on npm don't include this data and are much higher level (focusing on converting billing data and directly connecting to cloud providers)

Validations

N/A

Inputs

provider: The name of the cloud provider
instance-type: The instance type of the VM offering on that cloud provider.
location: The location where the VM is based
duration: Fraction of a day the VM is used for.
utilization: Average CPU utilization for that duration

Methodology

We will have to build our own model using the raw data from the CCF coefficients repo.
This gives the min/max watts per microarchitecture (per vCPU I presume) https://github.com/cloud-carbon-footprint/ccf-coefficients/blob/main/output/coefficients-aws-use.csv
This gives it seems a breakdown of all the instance types in AWS and some values for the powercurve for each instance type: https://github.com/cloud-carbon-footprint/ccf-coefficients/blob/main/data/aws-instances.csv
These scripts might give some indication regarding how the calculations are done. https://github.com/cloud-carbon-footprint/ccf-coefficients/tree/main/ccfcoef

Example IMPL

- component: 
    model: 
      path: ccf.cloud.sci
    config: 
      provider: aws
      instance_type: r6g.medium
    observations:
      series:
          -   datetime: 2023-07-06T00:00
              duration: 15 # translated into days_use_time
              cpu: 0.34  # translated into time_workload                          
          -   datetime: 2023-07-06T00:15
              duration: 15                              
              cpu: 0.12                                    
          -   datetime: 2023-07-06T00:30
              duration: 15                               
              cpu: 0.01                                     
          -   datetime: 2023-07-06T00:40
              duration: 15                               
              cpu: 0.78

Observations

Will be in the format of CPU utilization, durations.

Outputs

ImpactMetric (SCI style) will look like so for now:

{
 'e': xxx
 'm': xxx
}

Unit Tests

Calling with observations calls the API and returns a value for e and m.
Call with a single observation returns a single impact metric with the same timestamp and duration.
Call with multiple observations returns a multiple impact metrics with the same timestamp and duration as the original observation.
If you don't provide mandatory inputs it returns an error (please decide what the mandatory inputs are in terms of config and observations).
When the Boavizta API doesn't work (400/500 whatever) it returns an error.
If you provide an instance type that isn't supported, returns an error
If you provide a provider that isn't supported returns an error.
If you provide a region that isn't supported returns an error.

gnanakeethan commented 1 year ago

@jawache

I am planning to use the following information for the usage measurement.

AWS Instances - This has complete usage specification rather than just the information about per core. Such it is much easier to use this information against the data provided in the IMPL format.

AWS Calculated Coefficients - This link gives usage per vCPU according to the CPU Type for such, can only measure usage from the CPU part and not the rest.

jawache commented 1 year ago

Copying @gnanakeethan's comment from https://github.com/Green-Software-Foundation/carbon-ql/issues/50

https://www.npmjs.com/package/@cloud-carbon-footprint/core implements the calculation algorithms.

Following three packages include the coefficients to be used and initialized.

https://www.npmjs.com/package/@cloud-carbon-footprint/aws https://www.npmjs.com/package/@cloud-carbon-footprint/azure https://www.npmjs.com/package/@cloud-carbon-footprint/gcp

It might require a data-type that features information like which platform / cloud, then the usage metrics.

These packages can not be called directly but they need to be initialized and then called.

gnanakeethan commented 1 year ago

@jawache the data files for gcp does not have the information about the curve.

We will have to rely on the output information to further.

I will try to dissect how CCF currently utilises these output values.

gnanakeethan commented 1 year ago

@jawache Regarding the SCI calculations, I might have to include the emissions data.

Should I use the emissions information within the CCF data repository or I should use the generic one you have previously mentioned?

gnanakeethan commented 1 year ago

@jawache

CCF vs Boavizta Cloud API return completely different values.

Is it normal?

jawache commented 1 year ago

@jawache Regarding the SCI calculations, I might have to include the emissions data.

Should I use the emissions information within the CCF data repository or I should use the generic one you have previously mentioned?

Ignore those we will have a separate source, just return energy for now then rest will fit in automatically.

gnanakeethan commented 1 year ago

@jawache

I have confirmed the following information for "AWS" - "t2.micro" instance type - "1hour usage"

Boavizta outputs:

{"e": 0.022222222222222223, "m": 34,000 }

CCF outputs:

{ "e": 0.004900000000000001, "m": 1,477,540 }

"m" value is kind of weird since it has to be constant.

Embodied emissions for each family is same regardless of the size in "CCF" output. https://github.com/cloud-carbon-footprint/ccf-coefficients/blob/main/output/coefficients-aws-embodied.csv

For Boavizta, it changes according to the instance family & size as well.

jawache commented 1 year ago

CCF has the total embodied emissions, need to scale it to the request using the below equation (by time-share and by resource-share)

Where:

TE = Total Embodied Emissions, the sum of Life Cycle Assessment(LCA) emissions for all hardware components TR = Time Reserved, the length of time the hardware is reserved for use by the software EL = Expected Lifespan, the anticipated time that the equipment will be installed RR = Resources Reserved, the number of resources reserved for use by the software. TR = Total Resources, the total number of resources available.

jawache commented 1 year ago

@gnanakeethan

[x] Complete the module to support CCF for each cloud provider.
[x] The unit tests above are new, this task won't be complete unless the unit tests are complete as per described above.

jawache commented 1 year ago

Closing as the associated PR is closed. https://github.com/Green-Software-Foundation/ief/pull/74

Green-Software-Foundation / if