Open mapmeld opened 2 months ago
Hi! That works for me, but I would add a more detailed hlog message at the resize step: if you're adding tokens, we need to know how many, since they are likely to behave weirdly, and if you're removing tokens from the vocab, we would want to know which ones. So a log like "You're using the adapter model's tokenizer, which has less/more tokens than its base: adding/removing X tokens" + in case of removing specifying which, would be useful.
You'll also need to run ruff to fix the style.
Re your comment, adapter_weights
and delta_weights
should be skippable by default, what's the error message you're getting when not specifying them?
I rebased, reformatted, and created a smaller demo to trigger the adding/removing token messages https://colab.research.google.com/drive/1iZhdL-HTuStKGZ3ezspd53eDt_lhXbvm
Unfortunately I can't test due to a Torch/TorchVision compatibility issue. If I install and run lighteval on CoLab I get this error:
RuntimeError: Failed to import transformers.models.bert.modeling_bert because of the following error (look up to see its traceback):
operator torchvision::nms does not exist
After upgrading torch and torchvision, I get:
File "/content/lighteval/src/lighteval/data.py", line 28, in <module>
from torch.utils.data.distributed import DistributedSampler, T_co
ImportError: cannot import name 'T_co' from 'torch.utils.data.distributed' (/usr/local/lib/python3.10/dist-packages/torch/utils/data/distributed.py)
Hi! Yes, you can't use the latest of pytorch, it's got a breaking change.
OK ✔️ I changed how the dependencies are installed and was able to run lighteval on the larger and smaller tokenizers
Updated CoLab, same URL: https://colab.research.google.com/drive/1iZhdL-HTuStKGZ3ezspd53eDt_lhXbvm
At the end of the notebook, I have added the error if I leave out delta_weights: false
from the config
File "/usr/local/lib/python3.10/dist-packages/lighteval/models/model_config.py", line 439, in create_model_config
if config["merged_weights"]["delta_weights"]:
KeyError: 'delta_weights'
I added a commit to this PR to change the line to if config["merged_weights"].get("delta_weights", False):
and the same for where it checks adapter_weights
PEFT has a feature for adapters to add tokens to a model: https://github.com/huggingface/peft/blob/main/examples/causal_language_modeling/peft_lora_clm_with_additional_tokens.ipynb
When using an AdapterModel with new tokens in LightEval, the script fails because:
AdapterModel._create_auto_tokenizer
always uses the base model path, without checkingconfig.tokenizer
Notebook with error: https://colab.research.google.com/drive/1AMJ6_MiZGFTBf8KdRn-zj7soKyZrzpbf?usp=sharing
This PR would create the tokenizer from
config.tokenizer or config.base_model
and runbase.resize_token_embeddings(...)
beforePeftModel.from_pretrained(base, adapter_weights)
This is based on my fix for llm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness/pull/1828Notes:
model_config_path
to AdapterModel and this is necessary for most advanced modelsadapter_weights
anddelta_weights
should be optional so I do not needdelta_weights: false
in my config