[Bug] Scheduler MedianStoppingRule is no impact

[sungreong]

You need to use custom metrics. But it doesn't seem to work well. In the whole iteration, it seems that the middle stop phenomenon is not detected and cannot be checked.

## example
def metric_sklearn(y_true= None, y_pred=None ):
      return np.mean(y_true-y_pred)

from sklearn.metrics import make_scorer
score = make_scorer(metric_sklearn, greater_is_better=False)
scheduler = MedianStoppingRule(grace_period=1.0,
metric="mean_test_score",mode="max",min_samples_required  =3.0)
param_space = {
    "module__num_units_1": hp.uniformint("module__num_units_1", 50, 100),
    "module__num_units_2": hp.uniformint("module__num_units_2", 50, 98),
    "module__num_units_3": hp.uniformint("module__num_units_3", 20, 40)
}
random = TuneSearchCV(
    net_regr,
    param_space,
    search_optimization="hyperopt",
    early_stopping=scheduler,
    scoring={"score" : score},
    n_trials= args.n_trials,
    n_jobs = args.n_jobs,
    verbose  = 1,
    cv = 5,
    refit="score",
)
random.fit(train_x_np, train_y_np)

i define the metric name is "mean_test_score". i think it is not effect

my result is below.

please help me

and i change max_iters , it is worked.

but i have a question

I wonder the exact meaning of max_iters.

In a neural network, if epoch is set to 1000 and max_iters is set to 10, does it mean that one trial is performed 10 times?

## https://docs.ray.io/en/master/tune/tutorials/tune-sklearn.html
The early_stopping parameter allows us to terminate unpromising configurations. If early_stopping=True, TuneGridSearchCV will default to using Tune’s ASHAScheduler. You can pass in a custom algorithm - see Tune’s documentation on schedulers here for a full list to choose from. max_iters is the maximum number of iterations a given hyperparameter set could run for; it may run for fewer iterations if it is early stopped.

Try running this compared to the GridSearchCV equivalent, and see the speedup for yourself!

[richardliaw]

@sungreong can you please provide a full script that I can run?

[richardliaw]

max_iters = number of training iterations per model (think like SGD epochs).