A user should be able to specify a given time budget for experimentation/tuning and the script should automatically adjust its work to honor that time budget. This requires some careful though/measurement/design. An initial plan could be:
Measure the time it takes for a single VW pass on the data (this might change depending on the parameters, however).
Divide up the time budget across the different experimentation areas (learning rate, regularization, feature interaction, estimators). Should the division be equal? Which areas matter more?
Report the total time taken and best set of parameters found so far.
A user should be able to specify a given time budget for experimentation/tuning and the script should automatically adjust its work to honor that time budget. This requires some careful though/measurement/design. An initial plan could be: