search

timoschick / bertram

This repository contains the code for "BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Representations".
https://arxiv.org/abs/1910.07181
Apache License 2.0
62 stars 12 forks source link

Example script for downstream task #3

Open pribadihcr opened 4 years ago

pribadihcr commented 4 years ago

hi @timoschick,

Any plan to provide example code for the downstream tasks mentioned in the paper. Thanks.

timoschick commented 4 years ago

Hi @pribadihcr, yes, we definitely plan to release the example code. However, the code is still kind of messy and I am really busy right now so it might take a few weeks :(

pribadihcr commented 4 years ago

Hi @timoschick,

Could you give me some hint the classification task's procedure?. e.g how to use bertram pretrained etc

timoschick commented 4 years ago

Sure, this is relatively straightforward. You can start from any of the examples found here, or you can use any other script that uses HuggingFace's Transformers for any downstream task. Before training/evaluating, you just need to load the BERTRAM model corresponding to the pretrained model used in the script and call:

 bertram.add_word_vectors_to_model(words_with_contexts, tokenizer, model)

where bertram is your BERTRAM model, tokenizer and model are the tokenizer and model used in your training/evaluation script, and words_with_contexts is a dictionary from rare words to a list of contexts in which they occur. For each rare word w, this will add a new token <BERTRAM:w> to the model's (and tokenizer's) embedding space. When processing examples for training/evaluation, you then simply need to replace each occurrence of a rare word w with <BERTRAM:w> (or with <BERTRAM:w> / w if you wish to use the "slash"-variant described in the paper).