Closed bokyeong1015 closed 5 months ago
AutoTokenizer
src/utils.py
src/lm_eval/models/huggingface.py
LlamaTokenizer
src/lora_retrain.py
tokenizer.pad_token: None
tokenizer.unk_token: None
Description
Changes
AutoTokenizer
for LLaMA-3 insrc/utils.py
andsrc/lm_eval/models/huggingface.py
LlamaTokenizer
is currently necessary for LLaMA-1 experiments.src/lora_retrain.py
AutoTokenizer
, the LLaMA-3 tokenizer yieldstokenizer.pad_token: None
&tokenizer.unk_token: None