[Question] within zeno.ipynb, how did you arrive at optimal hyper parameters

garrowat / zenobot

AI proverb generator, trained on 2000-10000 proverbs using a FastAI, PyTorch LSTM model. Converted to run on CPU for inference.

https://zenobot.garrettwatson.io

Apache License 2.0

9 stars 0 forks source link

[Question] within zeno.ipynb, how did you arrive at optimal hyper parameters #1

Closed abhishekms1047 closed 5 years ago

abhishekms1047 commented 5 years ago

I saw that you are passing these hyper parameters for your neural networks . Can you please explain on how you arrived at it and how do you know that it is optimal.

TEXT = data.Field(lower=True, tokenize=list)
bs=64; bptt=8; n_fac=42; n_hidden=128

TEXT3 = data.Field(lower=True, tokenize=list)
bs=64; bptt=8; n_fac=42; n_hidden2=512

Is there any methodology like Bayesian optimization,Grid Search,Genetic algos?

garrowat commented 5 years ago

Hi, sure! I'm quite novice so I'm afraid I can't provide much in the way of methodology.

bs=64: this was used during part 7 of the fast.ai course, and seemed to work well enough for Zenobot. I might try bs=1 for prediction as I hear that works well
bptt=8: again, suggested by fast.ai
n_fac=42: same reason
n_hidden=512: once my dataset got larger, I found that increasing the hidden layers improved the model