[Feature] [tune] create a mlflow run name from config params

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

https://ray.io

Apache License 2.0

33.79k stars 5.75k forks source link

[Feature] [tune] create a mlflow run name from config params #23228

Open Rusteam opened 2 years ago

Rusteam commented 2 years ago

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

Create a unique run name (derived from search config) for mlflow runs

Use case

When analyzing MLflow runs auto-generated by MLflowLogger it would be useful have a unique name for each run. For instance to analyze a lineplot with certain metrics.

As of now, it's hard to tell which run (params) yield best results from this plot. Screen Shot 2022-03-16 at 15 27 18

Related issues

No response

Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

amogkam commented 2 years ago

Hey @Rusteam which Ray version are you using? And which logger are you using- the new MLflowLoggerCallback, or the older MLflowLogger?

Rusteam commented 2 years ago

I'm using the callback one. However I've found out that's not possible from mlflow side see here. What we can do in the meantime is to add a custom run name in the dict of tags.

stale[bot] commented 2 years ago

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

miannini commented 2 years ago

there is any progress on this? I'm having the same issue, using Ray Tune with ML Flow to track hyperparameter optimization experiments. the run_name is the save for all trials and there is no way to compare results

stale[bot] commented 1 year ago

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

Rusteam commented 1 year ago

MLflow now creates random names by default. I can still see this to be useful, and I can implement it if there's any interest in the feature.

krfricke commented 1 year ago

@Rusteam it would be great if you could contribute this! Ideally we would just use the same trial name we use in the overview table?

Rusteam commented 1 year ago

Sounds good. Let me have a look at it

Rusteam commented 1 year ago

has it not been already solved here ?