Closed attardi closed 3 years ago
Since the vocab for BERT is not built from scratch, the pad_index comes from the BERT vocab rather than a default value. https://github.com/yzhangcs/parser/blob/f002b2ca32b021afd75c9dc4f360b82b5b96bf2b/supar/utils/field.py#L112
Sounds good. I forgot to say that in modules/bert.py I do this:
self.pad_index = self.bert.config.pad_token_id
It should be the same value, if pad is passed correctly to init(): which I wouldn't trust. In fact, I would eliminate the optional parameter.
The code uses two values for padding: 0, the default, for nn.Embedding such as char and tag feature types, and feat_pad_index for CharLSTM and Bert Embedding. A value for pad_index is stored as self.pad_index in the model BiaffineDependencyModel as well as saved among the parameters in a model file. But BERT models uses a different value, usually different from feat_pad_index and pad_index. In method BiaffineDependencyModel.forward(), self.pad_index is used to compute a mask:
but that might not be the right value. I run into this problem when using the French model camembert-base which uses a pad index of 1. Besides, in my experiments, I also get away with feature word_embed.
I propose is to use its own value on feats rather than on words: