search

google-research / bert

TensorFlow code and pre-trained models for BERT
https://arxiv.org/abs/1810.04805
Apache License 2.0
37.83k stars 9.56k forks source link

BERT vs Word2vec #362

Open vikaschib7 opened 5 years ago

vikaschib7 commented 5 years ago

Hello All,

Can you please help me out in getting similar words from the BERT model, as we do in Word2Vec?

Best regards, Vikas

TinaB19 commented 5 years ago

https://github.com/hanxiao/bert-as-service

crapthings commented 5 years ago

@Tina-19 is that repo just do encoding?

TinaB19 commented 5 years ago

Yes, it provides fixed-length vectors for sentences using BERT that can be used instead of Word2Vec.

vikaschib7 commented 5 years ago

Is it possible to see similar words in BERT like if I search for "radar sensor companies", so I can get the similar words related to above query @Tina-19 @andrewluchen @jacobdevlin-google

HoaiDuyLe commented 5 years ago

I think this link can help you. https://github.com/google-research/bert/issues/60

apogiatzis commented 5 years ago

One thing to realise is that word2vec provides context-free embeddings (static) whereas BERT gives contextualised embeddings (dynamic). For instance, take these two sentences "I like apples", "I like apple MacBooks". Word2vec will give the same embedding for the word apple in both sentences whereas BERT will give you a different one depending on the context.

Now coming back to your question, here is a step by step tutorial I wrote to obtain contextualised embeddings from BERT: https://towardsdatascience.com/nlp-extract-contextualized-word-embeddings-from-bert-keras-tf-67ef29f60a7b