Calculate perplexity for large language models

[VeritasJoker]

Use this script: https://github.com/hassonlab/247-pickling/blob/dev/scripts/tfsemb_perplexity.py

and this command: perp-embeddings: mkdir -p logs for conv_id in $(CONV_IDS); do \ python scripts/tfsemb_perplexity.py \ --project-id $(PRJCT_ID) \ --pkl-identifier $(PKL_IDENTIFIER) \ --subject $(SID) \ --conversation-id $$conv_id \ --embedding-type $(EMB_TYPE); \ done;

It should take less than a minute to run for each of those if we have the model downloaded

With stride 512, 1024, 2048, 4096:

• [x] gpt2-xl
• [x] EleutherAI/gpt-neox-20b
• [x] facebook/opt-6.7b
• [x] facebook/opt-13b
• [x] facebook/opt-30b
• [x] facebook/opt-66b
• [x] meta-llama/Llama-2-7b-hf
• [x] meta-llama/Llama-2-13b-hf
• [x] meta-llama/Llama-2-70b-hf

Actually can you just do the four strides for all of the models. I'm making a table for all the values here: https://docs.google.com/spreadsheets/d/1E3k9gCvqsWERyPmvXvo-0yfYIyFt5XJQhYNS2ykKooo/edit?usp=sharing

[hvgazula]

where did you find this script?

[VeritasJoker]

HuggingFace lol

Here: https://huggingface.co/docs/transformers/en/perplexity

[hvgazula]

Does it matter if the models are quantized or not?