BERT - Githubissues

clay-lab / transductions

A PyTorch framework for creating, running, and reproducing experiments on seq2seq models.

6 stars 3 forks source link

BERT #57

Open jopetty opened 3 years ago

jopetty commented 3 years ago

Would be nice to have BERT an option for the encoder. Some issues are:

BERT uses its own tokenizer, which probably doesn't play nicely with the Fields we've been using.
How do we get access to the source vocabulary
Do we always need to have the transformation token included in the target vocabulary, since BERT's tokenizer might do weird things to it?

jopetty commented 3 years ago

aedca6f adds a "working" (i.e., doesn't error) BERT model, but it doesn't seem to learn very well. Among the design considerations:

wiring up the actual config parameters in the YAML file so they do something. Not sure how much flexibility we have with the pre-trained HuggingFace models, but at the very least we shouldn't have extraneous options
it seems best (for training time) to freeze the layers of the BERT encoder, but maybe this should be a user-configurable option as well?

jopetty commented 3 years ago

It seems that the positional encodings built into the HuggingFace BERT models are not useful in a sequence to sequence context. I'm not really sure why this is, but it is fixable if we add our own positional encodings to the embedding layer of the pretrained models.