carbon footprint

[FynnBe]

Thanks @dasv74 for raising the issue of carbon footprints associated with models in the model zoo.

training

I propose to add an optional training_emissions field to the model RDF. This field should probably hold the CO2 equivalent in kg. We might want to differentiate a little more for the case of model refinement. Neither the accumulated, nor the refinement only value alone seem to be a fair measure. Subfields exlcluding_parent, including_parent or similar might be more informative. We can discuss this more over at bioimage-io/spec-bioimage-io if we decided here in principal to add this field.

inference

Like running the dynamic model test 'test inputs match test outputs' in bioimage-io/collection-biomage-io we can easily add co2 estimation for inference there. A typical image should be given by the test image. However, the result will likely greatly depend on the test image size and "punish" models that provide large test images. As most of our models accept varying input tensor sizes I propose to run inference for a (random) image as close in size as the model allows to some reference size for n times and then scale the result by a correction factor. Our reference size and the correction factor could be based on total tensor elements to allow a rough comparison across different model inputs. This, and in particular the question how/if we include multiple tensor inputs (or even outputs) needs some more thought... If we add any co2 estimates for inference though, they should be at least roughly comparable within the model zoo.

implmentation ideas to save/display inference emissions:

• "dynamic test": a test summary just like the static or dynamic tests generate
• entry in config:bioimageio:inference_emissions

CO2 estimation

The codecarbon python package seems very well suited straight forward to deploy (thanks @dasv74 for the suggestion). It estimates emissions for a code block:

from codecarbon import EmissionsTracker
tracker = EmissionsTracker()
tracker.start()
# GPU intensive training code
emissions = tracker.stop()

The output is documented here: https://mlco2.github.io/codecarbon/output.html

Fields | Description -- | -- experiment_id | A unique identifier using uuid library timestamp | Time of the experiment in %Y-%m-%dT%H:%M:%S format project_name | Name of the project, defaults to codecarbon duration | Duration of the compute, in seconds emissions | Emissions as CO₂-equivalents [CO₂eq], in kg energy_consumed | Total Power consumed by the underlying infrastructure, in kWh country_name | Name of the country where the infrastructure is hosted country_iso_code | 3-letter alphabet ISO Code of the respective country region | Province/State/City where the compute infrastructure is hosted on_cloud | Y if the infrastructure is on cloud, N in case of private infrastructure cloud_provider | One of the 3 major cloud providers, aws/azure/gcp cloud_region | Geographical Region for respective cloud provider,examples us-east-2 for aws, brazilsouth for azure, asia-east1 for gcp

Implementation detail for inference: Additionally to the online EmissionTracker, which fetches information about the current location (and thus the applicable power mix) online, the package provides an OfflineEmissionTracker which is initialized with the region information. We could use this to not track actual CO2 use, but instead CO2 equivalent for our region of choice to keep it more comparable. Besides the region the estimate will vary as the power consumption depends on the hardware (which we can't control if it runs on GitHub CI)

[FynnBe]

@oeway :

Besides that, I also noticed that other model zoos, such as the hugging face, are already integrating carbon footprint info, partly due to the fact that transformers are really bad at carbon emissions. Here is what they have: https://huggingface.co/docs/hub/model-repos#why-is-it-useful-to-calculate-the-carbon-emissions-of-my-model

[esgomezm]

Hi there! Thank you for opening the issue.

Can't we estimate the value of training_emissions with the test or inference? Or maybe it's too complicated to provide it as the value will really depend on each user's application? It will depend on the number of epochs, batch, data size, and whether there's any part froze in the architecture.

For the inference, maybe we can normalize by the size of the input image? Something like XX kg of CO2 for each pixel/voxel in the input image?

[FynnBe]

Can't we estimate the value of training_emissions with the test or inference? Or maybe it's too complicated [...]

Yes, I think this is not feasible. I suggest we keep training emissions optional and whoever actually trains a model can measure their "actual" (estimated based on the running training script) emissions and report it there.

For the inference, maybe we can normalize by the size of the input image? Something like XX kg of CO2 for each pixel/voxel in the input image?

Yes, exactly. Either counting across all non-batch channels (that's what I head in mind when writing "tensor elements") or excluding channels. I would not differentiate between spatial and channel dimensions here for simplicity (we also have the 'i' instance/index/element dimension).