search

automl / HPOBench

Collection of hyperparameter optimization benchmark problems
Apache License 2.0
136 stars 35 forks source link

MSE or "R"MSE in nas/tabular benchmarks? #184

Open Deathn0t opened 9 months ago

Deathn0t commented 9 months ago

Hello,

I noticed that in the original repo that generated the data here the mean_squared_err function is used.

But then, in the HPOBench the valid_rmse_... is used see here

After trying to reproduce a constant mean predictor (from the original regression dataset), at least for the parkinson telemonitoring benchmark, I have an estimated mse of about 0.9. (I did not find any sqrt call anywhere also) which is of the same order of magnitude as performances reported in the benchmark.

Do you have any guess about if it is mse or rmse?