Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
投稿日付(yyyy/MM/dd)
2018/10/11
概要
原文
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).
論文リンク
https://arxiv.org/abs/1810.04805
BERTのPyPI
https://pypi.org/project/pytorch-pretrained-bert/#usage
著者/所属機関
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
投稿日付(yyyy/MM/dd)
2018/10/11
概要
原文
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).
日本語でひとこと
とっくにまとめ記事があるので他のものをよんでいいと思うでよ。 NLPにおけるRepresentationを行うモデルBERTの提案論文。 BERT : Bidirectional Encoder Representations from Transformers. OpenAI GPTやELMoに競合して出した、2018末頃のSOTAをモデルで現在様々な形で用いられている一種のスタンダードとなったモデル。
新規性・差分
構造としてGPTに対しては双方向を、ELMoに対してはTransformer構造をとっている点が差分。
(提案部分でもある)NLPにおける分類・予測タスクで予め学習したBERTに１層追加して、特定タスクに合わせた転移学習を行うだけでかなりの精度を出せる。
手法
モデルアーキテクチャは、tensor2tensorのTransformerを用いた多層・双方向Transformer Encoder。 TransformerのEncoderをレイヤーとして用いている模様。
図の左手がBERT。(繰り返しだが)GPTに対しては双方向を、ELMoに対してはTransformer構造をとっている点が差分。
pre-train taskは２種
Masked LM 入力分を確率で一部マスクし、その部分を予測させる
Next Sentence Prediction (NSP) 連続した2文とそうでない2文を半々で出題し、連続しているかを予測させる
結果
コメント
余談だがBERTの出力自体をLanguage部分の特徴量として扱うことでVQAのような別タスクでも有用な特徴量として扱うことが可能。 汎用言語表現モデルBERTを日本語で動かす(PyTorch)