Add resize_token_embeddings feature

[ccdv-ai]

⚠️ Please check that this feature request hasn't been suggested before.

• [X] I searched previous Ideas in Discussions didn't find any similar feature requests.
• [X] I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

Add the option to resize the token embeddings: PreTrainedModel has this method.

✔️ Solution

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(model_id)
model.resize_token_embeddings(num_tokens)

❓ Alternatives

No response

📝 Additional Context

No response

Acknowledgements

• [X] My issue title is concise, descriptive, and in title casing.
• [X] I have searched the existing issues to make sure this feature has not been requested yet.
• [X] I have provided enough information for the maintainers to understand and evaluate this request.

[NanoCode012]

May I ask what is the use case? We currently resize to the tokenizer's length (or multiple of 32 to it if enabled).

[ccdv-ai]

Can be usefull for a tokenizer/custom tokenizer and vocab_size mismatch. For instance, there is a mismatch for all qwen 2.5 models where they padded the embedding layer for distributed training.

[NanoCode012]

@ccdv-ai , just to clarify, would you want a config that lets you specify the new tokenizer vocab size or just to resize?

Axolotl does the latter* under the hood when you add new tokens. https://github.com/axolotl-ai-cloud/axolotl/blob/8c3a727f9d60ffd3af385f90bcc3fa3a56398fe1/src/axolotl/utils/models.py#L1039-L1053

If you enable resize_token_embeddings_to_32x: true, it will resize to next multiple of 32.

[ccdv-ai]

@NanoCode012 Only the option to choose and resize the token_embeddings to an arbitrary value. For example, Qwen 2.5 7B tokenizer has 151665 tokens but the embedding layer has 152064. resize_token_embeddings_to: 151665 should be possible.

if self.cfg.resize_token_embeddings_to < len(self.tokenizer):
    #Warning or stop
self.model.resize_token_embeddings(self.cfg.resize_token_embeddings_to, **resize_kwargs)

[NanoCode012]

@ccdv-ai thanks for clarifying. To add on to your point, we already do resize to tokenizer.

# above code but summarized

embeddings_len = len(self.tokenizer) 

if ( 
     self.model.get_input_embeddings().num_embeddings < embeddings_len 
 ): 
     self.model.resize_token_embeddings(embeddings_len)

For resizing to another value (!=len(self.tokenizer) ) , I'm not sure I understand the use case as the tokenizer would then mismatch the embedding length and cause an error during training.