search

cooelf / SemBERT

Semantics-aware BERT for Language Understanding (AAAI 2020)
https://arxiv.org/abs/1909.02209
MIT License
286 stars 55 forks source link

Interesting contribution #2

Closed luisfredgs closed 4 years ago

luisfredgs commented 4 years ago

Your study is an interesting contribution. I have a shallow question: Do you have used BERT only as a tokenizer in a subword level? Anyway, as the source code will give more details about, i'll appreciate if you turn it avaliable for us.

Thank you for sharing you work.

cooelf commented 4 years ago

Yes, we used BERT only as the subword tokenizer. We will release the codes and the labeled data to help quick implement, hopefully in the next week.

luisfredgs commented 4 years ago

The manner you provide rich semantic embeddings is a great insight. Thanks for sharing the source code.

drussellmrichie commented 2 years ago

Yes, we used BERT only as the subword tokenizer.

Hi, @cooelf. I think I am a bit confused by this. The paper seems to imply that BERT is used not just to tokenize words into subwords, but also to get contextualized representations for the subwords. For example, you have this figure, which shows the interactions between tokens:

image

Then there is also the following passage in the paper (I added bolding):

The raw text sequences and semantic role label sequences are firstly represented as embedding vectors to feed a pre-trained BERT. The input sentence X = {x1, . . . , xn} is a sequence of words of length n, which is first tokenized to word pieces (subword tokens). Then the transformer encoder captures the contextual information for each token via self-attention and produces a sequence of contextual embeddings.

Can you clarify, @cooelf?