Training hidden_units of DNN with skflow

[vinhqdang]

Hello all

As I know, there are several ways to optimize DNN. I think the most important parameters for DNN should be hidden_units.

Is it possible to somehow find best hidden_units automatically?

[ilblackdragon]

You can try using GridSearchCV from sklearn (http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html). A (almost) example usage here: https://github.com/google/skflow/blob/master/skflow/tests/test_grid_search.py

Otherwise - it's really just an experiments and trying to understand how it changes the quality of the model. There just basic rules of thumb (like start with the same number as original features and then go less and less: e.g. for 500 features you can use [500, 250, 100]). You also want to balance the number of examples you have versus a number of parameters. Usually, you want to have ~10 examples per parameter.

[vinhqdang]

Haha, thank you for giving this rule of thumb.

I have to confess that there a lot "rule of thumb" presented in discussion in quora or stackoverflow or researchgate to construct hidden layers in DNN, but let me check with your rule.

[vinhqdang]

I think GridSearch in sklearn is similar with h2o.grid in h2o deep learning.

However, the idea is you need to present all the possible hidden_units. We can list 1 000 000 000 different hidden_units and let the computer run, but I am not sure if it is a smart way. (For me it is just a brute force approach - maybe I am wrong)

[ilblackdragon]

There are other hyperparameter optimizers that actually take into account the values and accuracy of the model to pick next values (we use something like this in Google). For example of Open Source library you can try - http://hyperopt.github.io/hyperopt/

[ilblackdragon]

Hope this helped! I'm closing this bug for now.