optuna / optuna-integration

Extended functionalities for Optuna in combination with third-party libraries.
https://optuna-integration.readthedocs.io/en/latest/index.html
MIT License
39 stars 30 forks source link

Add trial number to MLflowCallback log_params #110

Closed TTRh closed 7 months ago

TTRh commented 7 months ago

Motivation

In mlflow we have the capability to create charts based on hpo parameters and metrics, but we are missing the trial number as parameter to for instance create chart of metrics over trials.

Description of the changes

With this change we are logging the params just as before but we also add the trial number as parameter

nzw0301 commented 7 months ago

I'm not sure what do you mean. The trial numbers are stored as the names of trials, so we can create a charts over trials. Could you explain a little bit more?

nzw0301 commented 7 months ago

Actually, mlflow integration has such an interface as is
https://optuna-integration.readthedocs.io/en/v3.6.0/reference/generated/optuna_integration.MLflowCallback.html#optuna_integration.MLflowCallback.track_in_mlflow. By using this, we can call mlflow.log_param for any argument in the objective.

TTRh commented 7 months ago

Hi @nzw0301 thanks for the quick reply!

What i wanted is to be able to chart metrics over time (based on trial number), the default mlflow callback add the trial number as tags but not as param and we can't use tags in axis for chart in mlflow. Example of possible new chart:

Screenshot from 2024-04-08 18-04-29

But actually thanks for the link to the doc i definitly overlook the official documentation cause with the decorator i can actually add anything i want as param this is definitly a better solution. I'm closing this one then.

nzw0301 commented 7 months ago

Thank you for your clarification and glad to hear that the existing implementation works for your use case!

TTRh commented 7 months ago

@nzw0301 I tested the decorator everything works as expected :tada: i just spotted a small issue with run name and made a proposal here: https://github.com/optuna/optuna-integration/pull/111