Viewing and interpreting results

[yngtodd]

Thanks for putting together this library!

What is the best way to view and interpret the pickled results from one of the runs?

[zygmuntz]

As far as viewing goes, maybe that will do: https://github.com/zygmuntz/hyperband/blob/master/show_results.py

[yngtodd]

Great, thank you very much for writing that up!

[yngtodd]

These results are really interesting. I am curious about one thing though and am wondering if I messed something up.

In the problem that I was testing out, I loaded in the Boston housing prices dataset from Scikit-Learn, and was using the gb.py as a starting point to write a gb_regressor.py for my problem. I then wrote a function similar to train_and_eval_sklearn_regressor that takes in the gb_regressor and the Boston data, fits the model, and returns the negative mean absolute error over five fold cross validation. Here is how I specified that return value in my version of train_and_eval_sklearn_regresssor:

loss = -np.mean(cross_val_score(clf, x_train, y_train, cv=5,
                                scoring="neg_mean_absolute_error"))

Now here is where I am curious: when looking at the results running hyperband, I am finding that the loss is generally increasing over the many runs. Here are the final few runs of the optimization:

Am I correct in assuming that the bandit is trying to gain the least amount of loss over the long term, i.e. minimizing the loss over the many runs?

[zygmuntz]

There is already a gb regressor in defs_regression/. I would suggest you try running it and see how it behaves.

Hyperband uses random search so the loss goes up and down from run to run.