DE v.s. long time running gridsearch

[WeiFoo]

Expriment

• data set: training on release 1, tuning on release 2, testing on release3 (The same as int the journal paper)
• GridSearch:
  • CART: 5 parameters, each dimension is equally divided into 5, totally 5^5= 3105 evaluations
  • RF: 6 parameters, each dimension is equally divided into 5, totally 5^6= 15625 evaluations
  • DE: use np=50, life =5, using evaluations from 250 to 400

[WeiFoo]

Result

Precision

F-measure

[WeiFoo]

Comments

• CART: grid search and DE have the almost same precision, DE has a little better F score than grid search.
• RF: grid search has higher precision, but a little lower F score
• BUT, take evaluations/running time into account, DE might be better !

[WeiFoo]

Precision

F

[WeiFoo]

0321 update!!

with all grid search results.

with only significant grid search results.

[WeiFoo]

running time(in csv format)

here, just list them as csv to help reformat.

precision

Name,antV0,antV1,antV2,camelV0,camelV1,ivyV0,jeditV0,jeditV1,jeditV2,log4jV0,luceneV0,poiV0,poiV1,synapseV0,velocityV0,xercesV0,xercesV1
Naive_CART Running Time,1.462,2.008,3.315,4.251,5.658,1.692,2.149,2.419,2.826,1.103,1.924,2.338,2.805,1.555,1.611,2.537,3.506
Tuned_CART Running Time,54.394,86.264,105.875,168.929,278.809,59.729,111.209,114.194,124.253,40.087,81.736,100.806,126.876,62.542,74.124,105.941,153.272
Grid_CART Running Time,2334.395,3527.792,4744.795,6726.292,10358.937,2650.796,4305.939,4562.838,5015.191,1860.979,3355.516,4169.53,5224.447,2842.939,3058.87,4431.064,6473.253
Naive_RF Running Time,6.314,7.681,10.949,15.448,21.525,7.349,9.995,10.371,11.266,5.719,9.201,9.815,11.319,7.462,7.701,9.303,12.18
Tuned_RF Running Time,310.171,358.543,358.473,427.601,660.249,297.188,395.389,439.195,456.743,323.71,382.99,381.056,341.967,301.528,379.481,375.008,478.944
Grid_RF Running Time,64233.322,72097.073,81209.173,91192.92,124188.71,65999.631,80729.813,83188.876,85837.594,62348.186,73417.981,77980.934,84853.259,68130.04,70457.776,77891.418,95135.245

f-measure

Name,antV0,antV1,antV2,camelV0,camelV1,ivyV0,jeditV0,jeditV1,jeditV2,log4jV0,luceneV0,poiV0,poiV1,synapseV0,velocityV0,xercesV0,xercesV1
Naive_CART Running Time,1.463,1.991,3.321,4.288,5.665,1.663,2.15,2.465,2.944,1.087,1.963,2.317,2.835,1.548,1.571,2.441,3.448
Tuned_CART Running Time,55.573,84.86,124.63,183.818,233.707,61.547,98.331,103.205,118.259,40.849,76.051,105.572,126.373,70.579,66.841,103.715,151.34
Grid_CART Running Time,2341.352,3557.727,4766.674,6737.07,10455.416,2635.602,4295.584,4563.535,4997.232,1835.859,3312.521,4191.537,5271.798,2836.027,3071.883,4345.14,6361.658
Naive_RF Running Time,6.35,7.665,11.084,15.942,21.725,7.456,9.888,10.357,11.28,5.81,9.101,9.867,11.321,7.45,7.605,9.226,12.061
Tuned_RF Running Time,342.875,346.096,395.495,543.0,613.831,313.787,359.449,352.95,406.252,305.072,323.41,402.505,442.97,303.955,309.347,413.456,410.453
Grid_RF Running Time,64699.898,72916.557,81865.779,91661.111,124974.467,65756.946,80106.807,82560.228,85201.293,62312.746,72977.107,77909.892,84900.394,67671.196,70645.905,76614.252,93669.853

[WeiFoo]

@timm results are updated.

[timm]

aarggggg why do you give me raw csv and make your supervisor do the pretties. never again!

• [X ] question: in the following the runtimes were nearly identical tuning for F and tuning for precision. any real value in showing both? I think we could jsut show the following graphs

F

runtimes

time relative to running raw learner

[WeiFoo]

aarggggg why do you give me raw csv and make your supervisor do the pretties. never again!

not sure which kind of visualization you like :smile: .

• [ ] question: in the following the runtimes were nearly identical tuning for F and tuning for precision. any real value in showing both? I think we could jsut show the following graphs

I think no siginificant difference for running time between F and precision. those graphs for demonstration are enough

[timm]

wei- can you change the shape of your report graphs, 5" wide , y-axis stops at 80% (so no wasted white space on top), use mininum oissuble height of graph (where the key does not overrun the lines)

and change the key to "Untuned, DE, Grid"

and change the x-axis to "Data sets, sorted by results using default tunings."

thanks

[timm]

addit: NOT 5" wide. Can you make them 2.5 " wide? so they can sit side by side on top of page?

[WeiFoo]

I have upload the small graphs to sharelatex. Let me know if you want to make other changes.@timm