search

lfwa / carbontracker

Track and predict the energy consumption and carbon footprint of training deep learning models.
MIT License
352 stars 26 forks source link

Ignore incomplete log files when parsing #37

Closed raghavian closed 3 years ago

raghavian commented 3 years ago

Ignore log files that only have the predicted consumption if the run was aborted and the output log does not have the final consumption. This can happen in cases when the user sees a problem in the training and chooses to abort training. Currently, the parser throws an error and is unable to parse the directory of logs, even when it encounters a single incomplete log.

raghavian commented 3 years ago

Should avoid the cases where there is a missing output or standard log, instead of throwing an error. Exclude the mismatched logs perhaps.