issues
search
Nota-NetsPresso
/
shortened-llm
Compressed LLMs for Efficient Text Generation [ICLR'24 Workshop]
63
stars
8
forks
source link
Add scripts for Gemma
#6
Closed
bokyeong1015
closed
6 months ago
bokyeong1015
commented
6 months ago
Description
Add example scripts for Gemma-{2b, 7b}
Collaborated with @lifelongeeek; Related to #5
Changes
Add example scripts for Gemma-{2b, 7b}
Use
--add_bos_to_every
option for
src/anal_block_sensitivity_taylor.py
and
src/anal_block_sensitivity_ppl.py
To Do (not in this PR)
Update
src/lm_eval
for measuring commonsense reasoning accuracy
Gemma seems especially sensitive to the inclusion of a BOS token
PPL Results
Gemma-2b
| Model (Params) | Use `--add_bos_to_every`
for pruning criterion | Metric | PPL↓
WikiText2 | PPL↓
PTB | |---|:---:|:---:|:---:|:---:| | gemma-2b (2.5B) | n/a | ppl | 649.7 | 2969.3 | | **gemma-2b (2.5B)** | **n/a** | **ppl_bos** | **21.1** | **82.3** | | st-gemma-ppl (2B) | X | ppl | 295.2 | 1292.1 | | st-gemma-ppl (2B) | X | ppl_bos | 35.2 | 186.9 | | st-gemma-ppl (2B) | O | ppl | 524.1 | 3075.6 | | **st-gemma-ppl (2B)** | **O** | **ppl_bos** | **30.2** | **133.6** | | st-gemma-taylor (2B) | X | ppl | 38356.8 | 44494.7 | | st-gemma-taylor (2B) | X | ppl_bos | 83.9 | 321.6 | | st-gemma-taylor (2B) | O | ppl | 547.1 | 3458.0 | | **st-gemma-taylor (2B)** | **O** | **ppl_bos** | **29.3** | **149.0** |
Gemma-7b
| Model (Params) | Use `--add_bos_to_every`
for pruning criterion | Metric | PPL↓
WikiText2 | PPL↓
PTB | |---|:---:|:---:|:---:|:---:| | gemma-7b (8.5B) | n/a | ppl | 3119318.2 | 19714697.9 | | **gemma-7b (8.5B)** | **n/a** | **ppl_bos** | **17.2** | **62.8** | | st-gemma-ppl (7.1B) | X | ppl | 8680264.0 | 58857215.9 | | st-gemma-ppl (7.1B) | X | ppl_bos | 43.6 | 173.5 | | st-gemma-ppl (7.1B) | O | ppl | 17128426.3 | 124600727.1 | | **st-gemma-ppl (7.1B)** | **O** | **ppl_bos** | **21.8** | **88.6** | | st-gemma-taylor (7.1B) | X | ppl | 448704455.2 | 5135150558.7 | | st-gemma-taylor (7.1B) | X | ppl_bos | 580.1 | 2549.7 | | st-gemma-taylor (7.1B) | O | ppl | 16862874.7 | 147967024.6 | | **st-gemma-taylor (7.1B)** | **O** | **ppl_bos** | **22.4** | **89.0** |
Description
Changes
--add_bos_to_everyoption for src/anal_block_sensitivity_taylor.py and src/anal_block_sensitivity_ppl.pyTo Do (not in this PR)
src/lm_evalfor measuring commonsense reasoning accuracyPPL Results
Gemma-2b
| Model (Params) | Use `--add_bos_to_every`for pruning criterion | Metric | PPL↓
WikiText2 | PPL↓
PTB | |---|:---:|:---:|:---:|:---:| | gemma-2b (2.5B) | n/a | ppl | 649.7 | 2969.3 | | **gemma-2b (2.5B)** | **n/a** | **ppl_bos** | **21.1** | **82.3** | | st-gemma-ppl (2B) | X | ppl | 295.2 | 1292.1 | | st-gemma-ppl (2B) | X | ppl_bos | 35.2 | 186.9 | | st-gemma-ppl (2B) | O | ppl | 524.1 | 3075.6 | | **st-gemma-ppl (2B)** | **O** | **ppl_bos** | **30.2** | **133.6** | | st-gemma-taylor (2B) | X | ppl | 38356.8 | 44494.7 | | st-gemma-taylor (2B) | X | ppl_bos | 83.9 | 321.6 | | st-gemma-taylor (2B) | O | ppl | 547.1 | 3458.0 | | **st-gemma-taylor (2B)** | **O** | **ppl_bos** | **29.3** | **149.0** |
Gemma-7b
| Model (Params) | Use `--add_bos_to_every`for pruning criterion | Metric | PPL↓
WikiText2 | PPL↓
PTB | |---|:---:|:---:|:---:|:---:| | gemma-7b (8.5B) | n/a | ppl | 3119318.2 | 19714697.9 | | **gemma-7b (8.5B)** | **n/a** | **ppl_bos** | **17.2** | **62.8** | | st-gemma-ppl (7.1B) | X | ppl | 8680264.0 | 58857215.9 | | st-gemma-ppl (7.1B) | X | ppl_bos | 43.6 | 173.5 | | st-gemma-ppl (7.1B) | O | ppl | 17128426.3 | 124600727.1 | | **st-gemma-ppl (7.1B)** | **O** | **ppl_bos** | **21.8** | **88.6** | | st-gemma-taylor (7.1B) | X | ppl | 448704455.2 | 5135150558.7 | | st-gemma-taylor (7.1B) | X | ppl_bos | 580.1 | 2549.7 | | st-gemma-taylor (7.1B) | O | ppl | 16862874.7 | 147967024.6 | | **st-gemma-taylor (7.1B)** | **O** | **ppl_bos** | **22.4** | **89.0** |