search

lfwa / carbontracker

Track and predict the energy consumption and carbon footprint of training deep learning models.
MIT License
350 stars 27 forks source link

Print statements going in overdrive #70

Open andreasgoethals opened 3 months ago

andreasgoethals commented 3 months ago

When I run the CarbonTracker code, it always gives way too many print statements. Everytime you instantiate a tracker object it adds another line to the print statement, coming to this: CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring.

Snailed commented 2 months ago

Hello, thank you for reporting this issue. I experience it as well. I will take a look and try to fix this.