They analyze contextualized word representations from BERT.
2. What is amazing compared to previous works?
Previous work has found that BERT embeddings capture grammatical information (dependency tree) and it relates to Euclidean distance. -> This work analyzes more detail.
Previous works have analyzed embeddings from the Pipeline of BERT (pos tagging, coreference resolution, dependency labeling). -> This work analyzes internal representations from BERT.
3. Where is the key to technologies and techniques?
To analyze the grammatical information, they use model-wise attention vector.
4. How did evaluate it?
4.1 Grammatical information
Previous work found that the l2 distance between contextualized embeddings of BERT captures a dependency tree.
Figure 3 shows the average l2 distance of model-wise attention vector between two words with a given dependency label detects:
distant dependency such as parataxis (relation between the main verb of a clause and other sentential elements)
close dependency such as auxpass (passive information)
4.2 Internal representations
Figure 4 shows that BERT embeddings capture semantic information.
a1da4
commented
1 year ago
0. Paper
paper: arxiv
1. What is it?
They analyze contextualized word representations from BERT.
2. What is amazing compared to previous works?
3. Where is the key to technologies and techniques?
To analyze the grammatical information, they use model-wise attention vector.
4. How did evaluate it?
4.1 Grammatical information
Previous work found that the l2 distance between contextualized embeddings of BERT captures a dependency tree.
Figure 3 shows the average l2 distance of model-wise attention vector between two words with a given dependency label detects:
parataxis(relation between the main verb of a clause and other sentential elements)auxpass(passive information)4.2 Internal representations
Figure 4 shows that BERT embeddings capture semantic information.
5. Is there a discussion?
6. Which paper should read next?