Training data should be the same for both techniques, since GridSearch using cross evaluaiton with strafitied sampling, then we need to modify our DE traning data sets as well.
GridSearch:
Training data: release1+release2(2-fold CrossEvaluation with stratified sampling to split data.)
randomly pick 3 values for each parameter within given range(e.g:number_estimators:[68,90,133])
uniformly pick 3 values for each parameter within given range(e.g: nubmer_estimators:[50,100,150])
DE:
Training data: release1+release2(stratified sampling the data into new_train and new_tuning)
Testing data: release3
Results
Scores
Totally we have 17 experiments, the red numbers show the total number of highest scores occured between DE and GridSearch.
Running time:
Generally, DE is much faster than GridSearch, 5 times faster given current grid selection policy
DE usually run 60~100 evaluations, but GridSearch runs 2x3^4=162 evaluations for CART, 2x3^5 = 488 for RF(2 is because of 2-fold cross evaluation).
Settings
Training data should be the same for both techniques, since GridSearch using cross evaluaiton with strafitied sampling, then we need to modify our DE traning data sets as well.
Results
Scores
Totally we have 17 experiments, the red numbers show the total number of highest scores occured between DE and GridSearch.
Running time:
Generally, DE is much faster than GridSearch, 5 times faster given current grid selection policy DE usually run 60~100 evaluations, but GridSearch runs 2x3^4=162 evaluations for CART, 2x3^5 = 488 for RF(2 is because of 2-fold cross evaluation).
More details
Tuning goal: auc, GridSearch: randomly pick Tuning goal: auc, GridSearch: uniformly pick Tuning goal: precision, GridSearch: randomly pick Tuning goal: precision, GridSearch: uniformly pick Tuning goal: f1, GridSearch: randomly pick Tuning goal: f1, GridSearch: uniformly pick
Conclusion