Minibatches, dropout, fix Gamma, easier learning algorithm

better / convoys

Implementation of statistical models to analyze time lagged conversions

https://better.engineering/convoys/

MIT License

254 stars 42 forks source link

Minibatches, dropout, fix Gamma, easier learning algorithm #37

Closed erikbern closed 6 years ago

erikbern commented 6 years ago

This changes a bunch of stuff

Minibatches for faster convergence (although it makes every epoch a lot slower). This also gets rid of the checks for nan etc
Dropout rather than regression. Dropout on the inputs.
Fit an intercept to the regression model as a separate parameter just so that it's not affected by dropout
Turns out tf.igamma has a bug in the gradient wrt a (I'll open an issue with tensorflow later). Rather than setting it using gradient descent, do a minor perturbation after each step
Once gamma was fixed, turns out we can have a much simpler learning rate scheme, not save snapshots etc.

coveralls commented 6 years ago

Coverage increased (+0.6%) to 92.289% when pulling 3f15988c63826149362d7a3366496f574644021e on dropout into ef01b4be0b419500299b4e3dffd4d1c9bed8c515 on master.

coveralls commented 6 years ago

Coverage increased (+0.6%) to 92.289% when pulling 3f15988c63826149362d7a3366496f574644021e on dropout into ef01b4be0b419500299b4e3dffd4d1c9bed8c515 on master.

coveralls commented 6 years ago

Coverage increased (+0.6%) to 92.289% when pulling 41cf6e62935af0e530f962d974d3a4c9858ee589 on dropout into ef01b4be0b419500299b4e3dffd4d1c9bed8c515 on master.

erikbern commented 6 years ago

Code is a bit slower and a bit more bloated, but I've tested this extensively not just with unit tests but also real world applications, and it works much better.

Think I can clean up the regression models and remove a lot of redundancy also.

erikbern commented 6 years ago

Will do some more cleanup later