ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
32.01k stars 5.45k forks source link

Ray.train.report can only show four metrics in the console! other metrics wouldn't be displayed in the table #45611

Open youngwhite opened 1 month ago

youngwhite commented 1 month ago

What happened + What you expected to happen

when I use ray.tune to try the effects of different parameter compositions accross trials, I found that no matter how many metrics reported by 'train.report', it only shows four metrics in the console table.

Versions / Dependencies

ray 2.23.0

Reproduction script

import ray from ray import train, tune

def train_fn(config): metrics = { "loss": 1.0, "accuracy": 0.8, "precision": 0.75, "recall": 0.70, "f1_score": 0.72, # Additional metrics "lr": 111111111 } train.report(metrics)

return metrics

ray.init() reporter = tune.CLIReporter(max_column_length=100, infer_limit=10) reporter.add_metric_column('xxxxxxxxxxxxxxxxx') reporter.add_parameter_column('yyyyyyyyyyy')

tuner = tune.Tuner( train_fn, run_config=train.RunConfig(progress_reporter=reporter), tune_config=tune.TuneConfig(metric="loss", mode="min", num_samples=4) ) results = tuner.fit()

Issue Severity

Low: It annoys or frustrates me.

woshiyyya commented 1 month ago

Hi @youngwhite , can you share a snapshot of the table? By default there's a max_column_length that limits the width of the output table. We recommend you to find the full metrics dict in the final results, or report to experiment tracking tools like wandb and mlflow.

youngwhite commented 1 month ago

0 1 2

woshiyyya commented 1 month ago

I think the first few metrics already takes up the width of your CLI. Have you tried changing the order of the metrics and see if still 4 metrics are printed?

Again for checking a long list of metrics, I would suggest you to use experiment tracking tools for better visibility.

youngwhite commented 1 month ago

Yes, I thought of that too. Since I have a big screen for monitoring, I want to see more metrics during training. But it doesn't work even after I increased the max_column_length. That's pretty disappointing. Now I try to customize a logger by myself though it's troublesome.

youngwhite commented 1 month ago

I have tried to change the order of metrics, every time only the first four of them displayed.