search

Vahe1994 / SpQR

Apache License 2.0
515 stars 40 forks source link

Evaluation code for Falcon models #17

Open Amshaker opened 1 year ago

Amshaker commented 1 year ago

Hi Authors,

Thank you for sharing the evaluation code for llama models. Could you please release the code for the evaluation of Falcon models?

Best regards, Abdelrahman.

Vahe1994 commented 1 year ago

Hello! To run Falcon for perplexity calculation use code in https://github.com/Vahe1994/SpQR/blob/main/main_for_rw.py , it is similar to main.py with some additional code to allow run falcon. As for LM_eval - you can use lm_eval_main.py and provide falcon model in the args: --model hf-causal --model_args pretrained=tiiuae/falcon-40b. Feel free to report any problem that you encounter.

P.s. We are working on combining scripts for Falcon and LLama and deduplicating code, see PR https://github.com/Vahe1994/SpQR/pull/7 .We will add more explicit instruction how to run Falcon in ReadMe.