search

lfwa / carbontracker

Track and predict the energy consumption and carbon footprint of training deep learning models.
MIT License
382 stars 29 forks source link

AttributeError: 'CarbonTrackerThread' object has no attribute 'cur_epoch_time' #56

Closed mehdi-ericsson closed 1 year ago

mehdi-ericsson commented 1 year ago

Hi,

I wrote very simple callback function and used CarbonTracker in it. When I used it in a simple DL model, I faced with following error.

CarbonTracker: Traceback (most recent call last): File "/home/.../venv/lib/python3.10/site-packages/carbontracker/tracker.py", line 288, in epoch_end self.tracker.epoch_end() File "/home/.../venv/lib/python3.10/site-packages/carbontracker/tracker.py", line 152, in epoch_end self.epoch_times.append(time.time() - self.cur_epoch_time) AttributeError: 'CarbonTrackerThread' object has no attribute 'cur_epoch_time'

Callback function is also as follow. 

class CustomCallback(keras.callbacks.Callback):
    def on_train_begin(self,logs=None):
        self.tracker = CarbonTracker(epochs = 15)
        parser.parse_all_logs(log_dir="./carbon_emission_log/")

    def on_epoch_start(self, epoch, logs=None):
        self.tracker.epoch_start()

    def on_epoch_end(self, epoch, logs=None):
        self.tracker.epoch_end()

    def on_train_end(self, logs=None):
        self.tracker.stop()

Would you please let me know what is the problem?

Thanks.

PedramBakh commented 1 year ago

The cur_epoch_time attribute gets initialized when epoch_start() is called on the CarbonTracker object. It seems the time duration for each epoch in your case might have been too short for CarbonTracker to accurately capture. We've made updates in our recent release to address such scenarios by refining our tracking mechanism. Please take a look at the changes in Release v1.2.0. Given these enhancements, I'm proceeding to close this issue. Should you face any more challenges, feel free to open a new issue or share additional feedback.