search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.07k stars 5.6k forks source link

[Tune] Avoid insufficient resources warning if cluster is autoscaling #31292

Open bveeramani opened 1 year ago

bveeramani commented 1 year ago

Description

Don't raise warnings like:

2022-12-15 21:06:11,922 WARNING insufficient_resources_manager.py:133 -- Ignore this
message if the cluster is autoscaling. No trial is running and no new trial has been
started within at least the last 60.0 seconds. This could be due to the cluster not
having enough resources available to start the next trial. Stop the tuning job and
adjust the resources requested per trial (possibly via `resources_per_trial` or via
`num_workers` for rllib) and/or add more resources to your Ray runtime.

If the cluster is autoscaling.

Use case

Better UX.

Every time I train my model, I get bombarded with warning messages while I wait for my worker nodes to finish setting up. It's mildly annoying.

iojc commented 1 year ago

you can set log level to avoid warning

import ray.tune.execution.insufficient_resources_manager import logging

ray.tune.execution.insufficient_resources_manager.logger.warning("warning before setLevel") ray.tune.execution.insufficient_resources_manager.logger.setLevel(logging.CRITICAL) ray.tune.execution.insufficient_resources_manager.logger.warning("warning after setLevel")

terminal: warning before setLevel