分析类的文章。查看BERT对于structural inforamtion的学习能力。实验表明，在低层，BERT的phrasal representation捕捉到了phrase-level information。BERT’s intermediate layers encode a rich hierarchy of linguistic information(语言学信息), with surface features at the bottom(低层的表面特征？), syntactic features(中层的句法特征) in the middle and semantic features at the top(高层的语义特征).
In this work, we use probing tasks to assess individual model layers in their ability to
encode different types of linguistic features. We evaluate each layer of BERT using ten probing sentence-level datasets/tasks created by Conneau et al. (2018), which are grouped into three categories.
Surface tasks probe
sentence length (SentLen)
the presence of words in the sentence (WC)
Syntactic tasks test
sensitivity to word order (BShift)
the depth of the syntactic tree (TreeDepth)
the sequence of toplevel constituents in the syntax tree (TopConst).
Semantic tasks check
tense (Tense)
subject (resp. direct object) number in the main clause (SubjNum, resp. ObjNum),
the sensitivity to random replacement of a noun/verb (SOMO)
the random swapping of coordinated clausal conjuncts (CoordInv).
一句话总结：
分析类的文章。查看BERT对于structural inforamtion的学习能力。实验表明，在低层，BERT的phrasal representation捕捉到了phrase-level information。BERT’s intermediate layers encode a rich hierarchy of linguistic information(语言学信息), with surface features at the bottom(低层的表面特征？), syntactic features(中层的句法特征) in the middle and semantic features at the top(高层的语义特征).
资源：
论文信息：
笔记：
In this work, we use probing tasks to assess individual model layers in their ability to encode different types of linguistic features. We evaluate each layer of BERT using ten probing sentence-level datasets/tasks created by Conneau et al. (2018), which are grouped into three categories.
Surface tasks probe
Syntactic tasks test
Semantic tasks check
模型图：
结果：
接下来要看的论文：
这篇是ACL 2019的short，和我们的比较像。参考一下。