search

david-thrower / cerebros-core-algorithm-alpha

The Cerebros package is an ultra-precise Neural Architecture Search (NAS) / AutoML that is intended to much more closely mimic biological neurons than conventional neural network architecture strategies.
Other
27 stars 4 forks source link

Try--dropout-embedding-with-gpt-tokenizer-best-run #141

Open david-thrower opened 11 months ago

david-thrower commented 11 months ago

Kind of issue: kind/enhancement; R and D

Hybridize Alex's proof of concept for Droupout(0.75) -> Embedding(15 dimensions) with the parameters in f2fdcf708269fc9c9fd29ababd7b93cdc6f8f834

Suggested Labels (If you don't know, that's ok): kind/enhancement; R and D

david-thrower commented 11 months ago

@sashakolpakov

It looks like this approach Embedding -> dropout is a winner. I see that with the current params (that may no longer be optimal) I see test set binary-accuracy >= 0.95 with a dropout rate of ~ 0.6. I am testing on another branch, hybridizing this with the randomization of motivations and will see which of the 2 should be merged in.