Help run GPTFast on Mistral-7B-v0.1 (or v.02) for CausalML

MDK8888 / GPTFast

Accelerate your Hugging Face Transformers 7.6-9x. Native to Hugging Face and PyTorch.

Apache License 2.0

687 stars 65 forks source link

Help run GPTFast on Mistral-7B-v0.1 (or v.02) for CausalML #26

Open Rinocerbul opened 6 months ago

Rinocerbul commented 6 months ago

Hello,

I am having difficulties running GPTFast on Mistral-7b-v0.1, encountering the same errors as reported here: https://github.com/MDK8888/GPTFast/issues/25.

My assumption is that the model_config is not set properly (I am currently using the variant proposed in the llama example (https://github.com/MDK8888/GPTFast/blob/LlamaIntegration/Examples/llama.py).

Could you please help me out with the appropriate configuration?

Thank you!

MDK8888 commented 6 months ago

Hey, how are you? There's going to be an update for the Llama Model soon-there is a better way to do it than how it is currently done on that branch. I am very busy at the moment, but I will fix both Mistral and Llama in the next 3 weeks. :)