search

lisphilar / covid19-sir

CovsirPhy: Python library for COVID-19 analysis with phase-dependent SIR-derived ODE models.
https://lisphilar.github.io/covid19-sir/
Apache License 2.0
109 stars 44 forks source link

[New] change pruner of parameter estimation for speed-up #578

Closed lisphilar closed 3 years ago

lisphilar commented 3 years ago

Summary of this new feature

We use optuna package for parameter estimation. Documentation of optuna says "Pruners automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping)."

optuna provides some kinds of pruners as follows.

It will be useful to add pruner argument to Scenario.estimate() (internally, Estimator.run()). Then, if necessary, the default pruner should be changed.

Note: We can also change sampler (we use TPESampler, selectable from TPESampler, CmaEsSampler, GridSampler, RandomSampler). However, we should use TPESampler because the other samplers do not fit for our analysis.

lisphilar commented 3 years ago

As the first step, I added pruner argument to Scenario.estimate() (internally, Estimator.run()) with #579 (version 2.15.0-theta). Users will select pruners and keyword aruments of pruners as follows.

With my local PC and Japan data (-30Jan2021), Scenario.estimate(cs.SIRF, pruner="threshold", upper=0.5) (304 sec, RMSLE=0.2417) is better than Scenario.estimate(cs.SIRF, pruner="median") (as-is in version 2.15.0, 374 sec, RMSLE=0.2425).

Dear @Inglezos , I will check this comparison with Google Colab notebook, but do you have any ideas for parameter estimation speed-up?

lisphilar commented 3 years ago

With #585, Default value was changed from Scenario.estimate(cs.SIRF, pruner="median") to Scenario.estimate(cs.SIRF, pruner="threshold", upper=0.5) for speed-up.

Inglezos commented 3 years ago

Could hyperband pruner be more effective? As I see the threshold option was finally selected as default instead. Regarding the samplers, could NSGAIISampler and MOTPESampler be also considered?

lisphilar commented 3 years ago

Thank you for your reply.

Could hyperband pruner be more effective? As I see the threshold option was finally selected as default instead.

I forgot to write the result with hyperband, but threshold was better at that time. Results on my notebook are here.

  Time [sec] RMSLE
SuccessiveHalving 435 0.2417
Hyperband 325 0.2426
Median 374 0.2425
Threshold, upper=1.0 500 0.2514
Threshold, upper=0.5 304 0.2417
Threshold, upper=0.25 334 0.2425
Threshold, upper=0.1 506 0.2425
Percentile, percentile=75 346 0.2417

Regarding the samplers, could NSGAIISampler and MOTPESampler be also considered?

These samplers were added in the latest version of optuna and they are for multi-objective optimization (i.e. some taget scores are used for minimization/maximixation). We have only one score to minize (RMSLE) and our optimization is not a multi-objective optimzation.