search

mlco2 / codecarbon

Track emissions from Compute and recommend ways to reduce their impact on the environment.
https://mlco2.github.io/codecarbon
MIT License
1k stars 157 forks source link

EmissionTracker failed on GCP Colab Enteprise #572

Closed cl3m3nt closed 1 week ago

cl3m3nt commented 2 weeks ago

Description

Tried to launch codecarbon EmissionTracker on GCP Colab Enterprise. It works correctly on Colab Web with same codecarbon version 2.4.2. It failed on different GCP region trials always with same error message below. As you recently updated GCP location metatadata this might be related. I appreciate your help.

What I Did

from codecarbon import EmissionsTracker
tracker = EmissionsTracker()

[/usr/local/lib/python3.10/dist-packages/codecarbon/emissions_tracker.py](https://localhost:8080/#) in __init__(self, project_name, measure_power_secs, api_call_interval, api_endpoint, api_key, output_dir, output_file, save_to_file, save_to_api, save_to_logger, logging_logger, save_to_prometheus, prometheus_url, gpu_ids, emissions_endpoint, experiment_id, experiment_name, co2_signal_api_token, tracking_mode, log_level, on_csv_write, logger_preamble, default_cpu_power, pue)
    379         self._data_source = DataSource()
    380 
--> 381         cloud: CloudMetadata = self._get_cloud_metadata()
    382 
    383         if cloud.is_on_private_infra:

[/usr/local/lib/python3.10/dist-packages/codecarbon/emissions_tracker.py](https://localhost:8080/#) in _get_cloud_metadata(self)
    881     def _get_cloud_metadata(self) -> CloudMetadata:
    882         if self._cloud is None:
--> 883             self._cloud = CloudMetadata.from_utils()
    884         return self._cloud
    885 

[/usr/local/lib/python3.10/dist-packages/codecarbon/external/geography.py](https://localhost:8080/#) in from_utils(cls)
     44 
     45         provider: str = cloud_metadata["provider"].lower()
---> 46         region: str = extract_region_for_provider.get(provider)(cloud_metadata)
     47         if provider in ["aws", "azure"]:
     48             logger.warning(

[/usr/local/lib/python3.10/dist-packages/codecarbon/external/geography.py](https://localhost:8080/#) in <lambda>(x)
     35             "aws": lambda x: x["metadata"]["region"],
     36             "azure": lambda x: x["metadata"]["compute"]["location"],
---> 37             "gcp": lambda x: extract_gcp_region(x["metadata"]["zone"]),
     38         }
     39 

KeyError: 'zone'

@eherra

benoit-cty commented 2 weeks ago

Thanks for reporting this. Can you run the command curl http://169.254.169.254/computeMetadata/v1/instance/?recursive=true&alt=json because I expect that there is a missing field "zone" in "metadata". It is not related to the update of GCP location data in CodeCarbon. GCP may have modify the output of their API we use to know if we are on GCP and where.

cl3m3nt commented 2 weeks ago

Hi Benoit,

thanks for your reply. After I ran the command from GCP colab entreprise notebook I got the following:

Missing required header: Metadata-Flavor

I appreciate your feedback.

benoit-cty commented 2 weeks ago

Sorry, from https://stackoverflow.com/questions/62026424/is-it-possible-to-obtain-instance-metadata-for-gcp-on-google-collaboratory it seems to be 

curl -v -w "\n" -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/?recursive=true&alt=json
cl3m3nt commented 2 weeks ago

Hi,

here is the output:

benoit-cty commented 2 weeks ago

Sorry, it's curl -v -w "\n" -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/?recursive=true&alt=json

I tested it and I get this from Cloud Shell (I delete un-relevant lines)

{
  "cpuPlatform": "Intel Broadwell",
  "machineType": "projects/8xxxxx2/machineTypes/e2-standard-4",
  "zone": "projects/8xxxx2/zones/europe-west1-b"
}
cl3m3nt commented 2 weeks ago

Hi again,

tx for your reply. Not sure your test makes sense as it's comparing output from one environment being colab enteprise, the other being cloud shell. I appreciate your feedback.

Testing it from gcp colab enteprise notebook:

Testing from gcp cloud shell, got info like (deleted others lines):

"cpuPlatform": "Intel Broadwell", "description": "", "disks": [ { "deviceName": "boot", "index": 0, "interface": "SCSI", "mode": "READ_WRITE", "type": "PERSISTENT-BALANCED" }, { "deviceName": "home", "index": 1, "interface": "SCSI", "mode": "READ_WRITE", "type": "PERSISTENT" }

benoit-cty commented 2 weeks ago

So it seems that on Colab Entreprise you get an answer from the GCP API but it's an empty JSON {}

We have to update our GCP detection to handle this case. Thanks for reporting the details.

cl3m3nt commented 1 week ago

Hi,

tx for your feedback. Our team is looking forward to getting the fix. Kind regards

benoit-cty commented 1 week ago

Hello, I've opened a PR for that : https://github.com/mlco2/codecarbon/pull/580