Open Anindyadeep opened 1 year ago
gpu_layers
for non-llama/falcon models.gpu_layers
means number of layers to run on GPU. Depending on how much GPU memory is available you can increase gpu_layers
. Start with a larger value gpu_layers=100
and if it runs out of memory, try smaller values.stream=True
:
for text in llm(prompt, stream=True):
print(text, end="", flush=True)
That is awesome, also @marella I have one last question after which we can close this issue, can we convert a hugging face model to ggml through CTransformers. What I meant to say, suppose I took a llama 2 model and fine tuned it using peft and now I have the peft weights and I attached them to my hugging face model to work along, but now I want to change it to ggml, can I do through CTransformer interface.
Thanks
That is awesome, also @marella I have one last question after which we can close this issue, can we convert a hugging face model to ggml through CTransformers. What I meant to say, suppose I took a llama 2 model and fine tuned it using peft and now I have the peft weights and I attached them to my hugging face model to work along, but now I want to change it to ggml, can I do through CTransformer interface.
Thanks
Taking the liberty of linking this since your question is in two different issues: yes, cf https://github.com/marella/ctransformers/issues/67#issuecomment-1665217227
Yeah and thanks for clarifying and we can close this issue. Thanks
Recently CUDA support (experimental) was added for MPT models. Please update, try it out and let me know if the performance has improved.
Sure, let me check that out
@marella is there any way to find out how many gpu layers does a specific GPU has ?
I started out with experimenting a bit with CTransformers. The device I have been using is
And I tried to install
mpt-7B chat
ggml file and this was the codeFirst of all, I am not sure, whether it is using full GPU capacity and also the time taken was around 70 seconds and also I don't know the generation is not much good. Here is an example of the model response
====================== Model response ======================
====================== Model response end ======================
Although the model response is not perfect here. But if I keep that onto a side note then also the time require for all the generations are within the range of 60 - 80 seconds. So I have some of the following questions
gpu_layers
exactly means, I given values like 1, 3, 50 but getting not much of a difference