`dora grid ... -t 0` crashes if the job hasn't logged anything yet

facebookresearch / dora

Dora is an experiment management framework. It expresses grid searches as pure python files as part of your repo. It identifies experiments with a unique hash signature. Scale up to hundreds of experiments without losing your sanity.

MIT License

269 stars 24 forks source link

`dora grid ... -t 0` crashes if the job hasn't logged anything yet #40

Open louismartin opened 2 years ago

louismartin commented 2 years ago

🐛 Bug Report

When I run dora grid with the -t parameter to monitor the logs, the command stops if the job has not logged anything (e.g. it's still pending). It might be worth to continue monitoring even when the log does not exists yet.

$ dora grid my_grid -t 0
WARNING: Log .../dora_outputs/xps/d3843038/latest/61700001_0_log.out does not exist

adefossez commented 2 years ago

I agree this would be a good idea