search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.97k stars 5.77k forks source link

Ray.train.report can only show four metrics in the console! other metrics wouldn't be displayed in the table #45611

Open youngwhite opened 5 months ago

youngwhite commented 5 months ago

What happened + What you expected to happen

when I use ray.tune to try the effects of different parameter compositions accross trials, I found that no matter how many metrics reported by 'train.report', it only shows four metrics in the console table.

Versions / Dependencies

ray 2.23.0

Reproduction script

import ray from ray import train, tune

def train_fn(config): metrics = { "loss": 1.0, "accuracy": 0.8, "precision": 0.75, "recall": 0.70, "f1_score": 0.72, # Additional metrics "lr": 111111111 } train.report(metrics)

return metrics

ray.init() reporter = tune.CLIReporter(max_column_length=100, infer_limit=10) reporter.add_metric_column('xxxxxxxxxxxxxxxxx') reporter.add_parameter_column('yyyyyyyyyyy')

tuner = tune.Tuner( train_fn, run_config=train.RunConfig(progress_reporter=reporter), tune_config=tune.TuneConfig(metric="loss", mode="min", num_samples=4) ) results = tuner.fit()

Issue Severity

Low: It annoys or frustrates me.

woshiyyya commented 5 months ago

Hi @youngwhite , can you share a snapshot of the table? By default there's a max_column_length that limits the width of the output table. We recommend you to find the full metrics dict in the final results, or report to experiment tracking tools like wandb and mlflow.

youngwhite commented 5 months ago

0 1 2

woshiyyya commented 5 months ago

I think the first few metrics already takes up the width of your CLI. Have you tried changing the order of the metrics and see if still 4 metrics are printed?

Again for checking a long list of metrics, I would suggest you to use experiment tracking tools for better visibility.

youngwhite commented 5 months ago

Yes, I thought of that too. Since I have a big screen for monitoring, I want to see more metrics during training. But it doesn't work even after I increased the max_column_length. That's pretty disappointing. Now I try to customize a logger by myself though it's troublesome.

youngwhite commented 5 months ago

I have tried to change the order of metrics, every time only the first four of them displayed.