search

Breakend / experiment-impact-tracker

MIT License
266 stars 31 forks source link

Support for abruptly stopped experiments #37

Closed cifkao closed 3 years ago

cifkao commented 3 years ago

I interrupted an experiment and now when I run generate-carbon-impact-statement, it crashes with:

Traceback (most recent call last):
  File "venv/bin/generate-carbon-impact-statement", line 233, in <module>
    sys.exit(main(sys.argv[1:]))
  File "venv/bin/generate-carbon-impact-statement", line 177, in main
    extracted_info = gather_additional_info(info, log_dir)
  File "venv/lib/python3.6/site-packages/experiment_impact_tracker/utils.py", line 101, in gather_additional_info
    exp_len = datetime.timestamp(info["experiment_end"]) - \
KeyError: 'experiment_end'

I think this situation can be quite common, so the tracker should be able to recover from it (e.g. taking the last logged timestamp if experiment_end is not available) instead of crashing.

Breakend commented 3 years ago

This should be fixed in #51, we now allow the last logged timestamp as an estimate of the experiment length with a warning saying that this may not be accurate.