Xpose Word2Vec

[djinn-anthrope]

• [x] Word2Vec Naive Run
• [ ] W2V + Actual SGD (steal from GloVe)
• [x] W2V - 0 initialized + random initialized (Syn1Neg is random :P)
• [x] W2V where syn0 = syn1neg = random
• [x] Train cost function v.w, not sigm(v.w)
• [ ] Objective function: cos(theta), not sigm(v.w)

[bollu]

Another ablation that would be nice to have: word2vec without temperature tempering. That is, remove the code surrounding reducing alpha every iteration. Try this with our best model, and our baseline model.

In particular, the code to be killed would be:

            alpha = starting_alpha *
                    (1 - word_count_actual / (real)(iter * train_words + 1));
            if (alpha < starting_alpha * 0.001)
                alpha = starting_alpha * 0.001;

This would need to re-hyperparameter-tune alpha, since it was probably chosen by tuning in the first place...