optuna / optuna-dashboard

Real-time Web Dashboard for Optuna.
https://optuna-dashboard.readthedocs.io/en/latest/
Other
503 stars 83 forks source link

Add PED-ANOVA as the first option for importance evaluator #811

Closed nabenabe0928 closed 6 months ago

nabenabe0928 commented 6 months ago

Contributor License Agreement

This repository (optuna-dashboard) and Goptuna share common code. This pull request may therefore be ported to Goptuna. Make sure that you understand the consequences concerning licenses and check the box below if you accept the term before creating this pull request.

Reference Issues/PRs

This PR aims to add PED-ANOVA, a quick f-ANOVA calculation algorithm, to the first choice of the importance calculation of Optuna Dashboard to enhance UX.

For PED-ANOVA, please check the following PR:

What does this implement/fix? Explain your changes.

This PR introduces PED-ANOVA to Optuna Dashboard if the Optuna version in the environment is 3.6 or later. By using PED-ANOVA, the visualization becomes very quick, so users will not experience very lagged dashboard behavior anymore due to the importance evaluation.

The current Optuna Dashboard would not give anything after several minutes when n_trials is huge (e.g., n_trails > 10000).

image

Meanwhile, if we use PED-ANOVA, Optuna Dashboard gives the importance plot in a few seconds even with n_trails > 10000.

image

codecov[bot] commented 6 months ago

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (8f39413) 69.53% compared to head (3efee74) 69.53%. Report is 6 commits behind head on main.

Files Patch % Lines
optuna_dashboard/_importance.py 71.42% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #811 +/- ## ======================================= Coverage 69.53% 69.53% ======================================= Files 35 35 Lines 2360 2367 +7 ======================================= + Hits 1641 1646 +5 - Misses 719 721 +2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

nabenabe0928 commented 6 months ago

@c-bata Thank you for your review, I addressed your suggestion and removed the error in CI.