search

tensorflow / nmt

TensorFlow Neural Machine Translation Tutorial
Apache License 2.0
6.35k stars 1.96k forks source link

default embedding #472

Open nashid opened 4 years ago

nashid commented 4 years ago

If we do not provide embedding like word2vec, how does it know to represent the words?

Does it use one hot encoding by default or ngram, CBOW, skip grams?

luozhouyang commented 4 years ago

No. If you do not provide the pretrained embeddings, it will create an trainable variable, and initialize it by some algorithm. When you train the model on your data, this variable will be updated too.

nashid commented 4 years ago

@luozhouyang I understand if we do not provide the pre-trained embedding it uses the default implementation of embedding in this framework.

However, I would like to know what algorithm is used to build the embedding.

luozhouyang commented 4 years ago

Word embeddings here are actually an 2-d tensor, with shape (vocab_size, embedding_size). This tensor will be updated along with other params by BP.

nashid commented 2 years ago

@luozhouyang I understand this. But what algorithm it is using (like word2vec, gloVe, ...)?

luozhouyang commented 2 years ago

No special algorithm is used. Not word2vec, not GloVe, just a learnable 2-d matrix.