lfwa / carbontracker

Track and predict the energy consumption and carbon footprint of training deep learning models.
MIT License
397 stars 30 forks source link

Predicted CO2eq: nan g #18

Closed leondz closed 4 years ago

leondz commented 4 years ago

For trivial "hello world" jobs, actual consumption is reported but predicted consumption isn't:

#!/usr/bin/env python3
import time
from carbontracker.tracker import CarbonTracker
max_epochs = 1
tracker = CarbonTracker(epochs=max_epochs)
for epoch in range(max_epochs):
    tracker.epoch_start()
    time.sleep(20)
    tracker.epoch_end()
tracker.stop()

gives:

leon@blade:~/dev/lowtest$ python3 ct.py 
CarbonTracker: 
Actual consumption for 1 epoch(s):
    Time:   0:00:20
    Energy: 0.000263 kWh
    CO2eq:  0.019182 g
    This is equivalent to:
    0.000159 km travelled by car
/usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:3335: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/usr/local/lib/python3.7/dist-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
CarbonTracker: 
Predicted consumption for 1 epoch(s):
    Time:   0:00:20
    Energy: 0.000263 kWh
    CO2eq:  nan g
CarbonTracker: Finished monitoring.
leon@blade:~/dev/lowtest$ 
kanding commented 4 years ago

Hey Leon, thanks a lot for trying out the tool!

The API we use for Denmark ( https://www.energidataservice.dk/ ) supports predicting the forecast carbon intensity but since the total duration is predicted to be so short it fails to return a co2eq prediction. In this case it queries the forecasted carbon intensity over the, in this case, next 20 seconds but the API has no meaningful response to this.

We'll look into fixing this by using the current intensity if the predicted one is too short.

leondz commented 4 years ago

Thanks. Yeah, a lot of ML applications involve one-off classifications - they're necessarily hard to measure. Do any of the tmrw APIs offer finer granularity? A fallback could be simple to use the actual CO2eq instead.

On Wed, 8 Jul 2020 at 16:20, kanding notifications@github.com wrote:

Hey Leon, thanks a lot for trying out the tool!

The API we use for Denmark ( https://www.energidataservice.dk/ ) supports predicting the forecast carbon intensity but since the total duration is predicted to be so short it fails to return a co2eq prediction. In this case it queries the forecasted carbon intensity over the, in this case, next 20 seconds but the API has no meaningful response to this.

We'll look into fixing this by using the current intensity if the predicted one is too short.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lfwa/carbontracker/issues/18#issuecomment-655550751, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA5YTUKB4UAP5YNIBKT2L3R2R6EDANCNFSM4OUSDVAA .

lfwa commented 4 years ago

The free (sign-up required) API from tmrow, co2signal.com, does not support forecasted intensities. Their other API, api.electricitymap.org, does but is sadly non-free. The last time that we reached out to them they referred us to the co2signal.com API requiring users to sign up for their own API key.

Most of the APIs for fetching the current carbon intensity (that we have looked at) seems to be updated in 10-15 minute intervals, however, we cannot say for certain for tmrow's APIs.

If you are not interested in a prediction but just the actual consumption, you can set epochs_before_pred=0 in the constructor.

leondz commented 4 years ago

Thanks, that's clarifiying. Yeah, already using epochs_before_pred=0. The fallback was just a suggestion for gracefully avoiding outputting NaN :)