Evaluate batchtools - Githubissues

cmu-delphi / exploration-tooling

tools for evaluating and exploring forecasters

Other

0 stars 0 forks source link

Evaluate batchtools #1

Closed dshemetov closed 1 year ago

dshemetov commented 1 year ago

Use batchtools to do covid forecast evaluation
Use batchtools to do flu forecast evaluation
Confirm that arrow/parquet interacts well with it

dsweber2 commented 1 year ago

~~see if chunk jobs will let us do forecaster dependency tree~~ They will not. If we want to do sequential jobs, we're probably going to want to use tags to say 1st wave (independent forecasters), second wave (ensembles), third wave (ensembles containing ensembles), etc.

dsweber2 commented 1 year ago

Major thing left to test is what happens with parallelization.

dsweber2 commented 1 year ago

A limitation is that summarizeExperiments won't print jobs with parameters that are lists. Those jobs will run fine however

dshemetov commented 1 year ago

Ended up going with targets because it supports a notion of job dependence and allows for parameter sweeps across lists of complex objects (such as different trainers).

Built out a prototype job with file targets
Built out a prototype forecaster target
Built out a prototype ensemble forecaster target
Built out a parameter sweep job using tarchetypes