search

lfwa / carbontracker

Track and predict the energy consumption and carbon footprint of training deep learning models.
MIT License
382 stars 29 forks source link

Print statements going in overdrive #70

Closed andreasgoethals closed 1 month ago

andreasgoethals commented 7 months ago

When I run the CarbonTracker code, it always gives way too many print statements. Everytime you instantiate a tracker object it adds another line to the print statement, coming to this: CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: The following components were found: GPU with device(s) Tesla V100-SXM2-16GB. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Epoch duration is too short for a measurement to be collected. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: WARNING - Failed to retrieve carbon intensity: Defaulting to average carbon intensity 357.2021 gCO2/kWh. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Average carbon intensity during training was 357.20 gCO2/kWh at detected location: Council Bluffs, Iowa, US. CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Actual consumption: Time: 0:00:00 Energy: 0.000000000000 kWh CO2eq: 0.000000000000 g This is equivalent to: 0.000000000000 km travelled by car CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring. CarbonTracker: Finished monitoring.

Snailed commented 6 months ago

Hello, thank you for reporting this issue. I experience it as well. I will take a look and try to fix this.

Snailed commented 1 month ago

Resolved in Carbontracker v2.0.0.