How to add additional block analysis data for other models

Nota-NetsPresso / shortened-llm

Compressed LLMs for Efficient Text Generation [ICLR'24 Workshop]

63 stars 8 forks source link

How to add additional block analysis data for other models #12

Closed botox-100 closed 3 months ago

botox-100 commented 3 months ago

I like your approach and already use your implementation for my master thesis. However I want to use your workflow also for other models (especially Mistral/Mixtral). How can I create the block analysis data that is needed to use your approach. Thanks Thomas

bokyeong1015 commented 3 months ago

Hi, thanks for your interest :)

Please refer to our example script for CatPPT-base, which is based on MistralForCausalLM. We would recommend changing the Hugging Face model name in the script and fixing bugs if needed.

CatPPT-base (based on MistralForCausalLM) bash script/prune_CatPPT_crit-ppl.sh bash script/prune_CatPPT_crit-taylor.sh

Mixtral-based models haven't been tried yet, but we will happily reach out to you if we start working on them.