nstrayer
commented
5 years ago 1
As a general style note I would attempt to break this into more chunks with annotations. Right now you have some really long hard to read blocks of code.
Did you look into tuning the kernel for your SVM? That is probably the single most important hyper-parameter for performance. I think you may find your results change if you explored another kernel like poly or radial basis.
While random search is great when grid search is infeasible due to high dimensionality in the case of these models, especially the logistic regression, grid search is the way to go.
By tuning the C parameter for logistic regression you should see pretty drastic changes in your performance.
-3
2
Since the data here are periodic in nature and seasonal we can take advantage of this and use the periodic covariance function in pymc: pm.gp.cov.Periodic(1, period=365, ls = ls). I think you'll find much better convergence in that case. The matern is super flexible but ultimately isn't the best for periodic data which is why you ended up needing so many inducing points here.
3
Again, in regards to the random hyperparameter search it's relatively inefficient here when we can pretty exhaustively explore the parameter space with grid search in not that long. Also, you can save yourself a lot of hassle and just use gridsearchCV.
To me, the 3d plots here don't really add anything. They're too noisey to get much information out of them. I'd simplify and plot multiple lines colored by feature number, x-axis as depth, and faceted by number of trees or something along those lines. That way you can make concrete statements about trends. Here I can't tell any difference between the plots.
Your use of F1 is good here but I think you miss some of the interesting hyperparameter interaction effects that give some insights into model performance.
-2
4
A good way of getting at if you've over or under fit is to plot the train and test accuracy trajectories as you increase the proportion of your training data you use to train. An over-fit model will rapidly shoot its training accuracy to 1 and then the testing will plateau. An under-fit model will plateau its training and testing rather quickly.
You will almost always see that your train accuracy is higher than your test one regardless so it's hard to say if you've over or under-fit with just that single datapoint.
-3
Opening thread for feedback on homework 3