DataCollatorForLanguageModeling を試す

[konumaru]

• https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm_no_trainer.py
• https://www.kaggle.com/rhtsingh/commonlit-readability-prize-roberta-torch-itpt?scriptVersionId=63560998

[konumaru]

• RobertaForMaskedLM というモジュールを使って学習する
• RobertaModel の last_hidden_state[:. 0, :] を入力にする
• tokenizer によって生成されるlabelsを元に CrossEntropyLoss を算出

[konumaru]

class RobertaLMHead(nn.Module):
    """Roberta Head for masked language modeling."""

    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)

        self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
        self.bias = nn.Parameter(torch.zeros(config.vocab_size))

        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
        self.decoder.bias = self.bias

    def forward(self, features, **kwargs):
        x = self.dense(features)
        x = gelu(x)
        x = self.layer_norm(x)

        # project back to size of vocabulary with bias
        x = self.decoder(x)

        return x

[konumaru]

https://huggingface.co/transformers/_modules/transformers/models/roberta/modeling_roberta.html#RobertaModel

[konumaru]

続きは #6 で