huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
845 stars 100 forks source link

Allow AdapterModels to have custom tokens #306

Open mapmeld opened 2 months ago

mapmeld commented 2 months ago

PEFT has a feature for adapters to add tokens to a model: https://github.com/huggingface/peft/blob/main/examples/causal_language_modeling/peft_lora_clm_with_additional_tokens.ipynb

When using an AdapterModel with new tokens in LightEval, the script fails because:

Notebook with error: https://colab.research.google.com/drive/1AMJ6_MiZGFTBf8KdRn-zj7soKyZrzpbf?usp=sharing

This PR would create the tokenizer from config.tokenizer or config.base_model and run base.resize_token_embeddings(...) before PeftModel.from_pretrained(base, adapter_weights) This is based on my fix for llm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness/pull/1828

Notes:

clefourrier commented 2 weeks ago

Hi! That works for me, but I would add a more detailed hlog message at the resize step: if you're adding tokens, we need to know how many, since they are likely to behave weirdly, and if you're removing tokens from the vocab, we would want to know which ones. So a log like "You're using the adapter model's tokenizer, which has less/more tokens than its base: adding/removing X tokens" + in case of removing specifying which, would be useful.

You'll also need to run ruff to fix the style.

clefourrier commented 2 weeks ago

Re your comment, adapter_weights and delta_weights should be skippable by default, what's the error message you're getting when not specifying them?

mapmeld commented 5 days ago

I rebased, reformatted, and created a smaller demo to trigger the adding/removing token messages https://colab.research.google.com/drive/1iZhdL-HTuStKGZ3ezspd53eDt_lhXbvm

Unfortunately I can't test due to a Torch/TorchVision compatibility issue. If I install and run lighteval on CoLab I get this error:

RuntimeError: Failed to import transformers.models.bert.modeling_bert because of the following error (look up to see its traceback):
operator torchvision::nms does not exist

After upgrading torch and torchvision, I get:

  File "/content/lighteval/src/lighteval/data.py", line 28, in <module>
    from torch.utils.data.distributed import DistributedSampler, T_co
ImportError: cannot import name 'T_co' from 'torch.utils.data.distributed' (/usr/local/lib/python3.10/dist-packages/torch/utils/data/distributed.py)
clefourrier commented 4 days ago

Hi! Yes, you can't use the latest of pytorch, it's got a breaking change.

mapmeld commented 4 days ago

OK ✔️ I changed how the dependencies are installed and was able to run lighteval on the larger and smaller tokenizers

Updated CoLab, same URL: https://colab.research.google.com/drive/1iZhdL-HTuStKGZ3ezspd53eDt_lhXbvm

At the end of the notebook, I have added the error if I leave out delta_weights: false from the config

File "/usr/local/lib/python3.10/dist-packages/lighteval/models/model_config.py", line 439, in create_model_config
    if config["merged_weights"]["delta_weights"]:
KeyError: 'delta_weights'

I added a commit to this PR to change the line to if config["merged_weights"].get("delta_weights", False): and the same for where it checks adapter_weights