search

allenai / allennlp

An open-source NLP research library, built on PyTorch.
http://www.allennlp.org
Apache License 2.0
11.76k stars 2.25k forks source link

Allow Embedding Extension for T5 Model #5478

Closed HarshTrivedi closed 2 years ago

HarshTrivedi commented 2 years ago

I’m using T5 model implemented in allennlp, and need to add extra special tokens to its vocabulary.

I’ve added extra tokens in the tokenizer in my reader with self._tokenizer.tokenizer.add_tokens(additional_tokens). But I also need to extend model’s token embeddings. Usually, when the transformer is loaded with PretrainedTransformerEmbedder, it’s taken care of by itself because of this line. I can also do it by invoking HF’s resize_token_embeddings manually, but, T5 object here doesn’t have this method on it. The T5 object here is allennlp's module, so doesn't have HF methods like resize_token_embeddings.

My current solution is to manually extend T5 embeddings and lm_head, but it'd be good to have native support for this in allennlp.

@dirkgr Assigning you based on the discussion on slack.

github-actions[bot] commented 2 years ago

@dirkgr this is just a friendly ping to make sure you haven't forgotten about this issue 😜

github-actions[bot] commented 2 years ago

@dirkgr this is just a friendly ping to make sure you haven't forgotten about this issue 😜

github-actions[bot] commented 2 years ago

@dirkgr this is just a friendly ping to make sure you haven't forgotten about this issue 😜

dirkgr commented 2 years ago

Fixed in https://github.com/allenai/allennlp/pull/5497.