Support LLaMA-3 - Githubissues

Nota-NetsPresso / shortened-llm

Compressed LLMs for Efficient Text Generation [ICLR'24 Workshop]

63 stars 8 forks source link

Support LLaMA-3 #7

Closed bokyeong1015 closed 5 months ago

bokyeong1015 commented 5 months ago

Description

Add example scripts for LLaMA-3
Collaborated with @lifelongeeek

Changes

Use AutoTokenizer for LLaMA-3 in src/utils.py and src/lm_eval/models/huggingface.py
- cf. LlamaTokenizer is currently necessary for LLaMA-1 experiments.
Use EOS_token for PAD_token for LLaMA-3 in src/lora_retrain.py
- related links: 1, 2
- cf. with AutoTokenizer, the LLaMA-3 tokenizer yields tokenizer.pad_token: None & tokenizer.unk_token: None