search

marella / ctransformers

Python bindings for the Transformer models implemented in C/C++ using GGML library.
MIT License
1.8k stars 137 forks source link

Hello, how to specific config of maxnewtoken? #114

Closed lucasjinreal closed 1 year ago

lucasjinreal commented 1 year ago

Is there a convenient way to do this?

cfg = AutoConfig() llm = AutoModelForCausalLM.from_pretrained(m_f, gpu_layers=150)

marella commented 1 year ago

Hi, you can pass it to from_pretrained():

llm = AutoModelForCausalLM.from_pretrained(..., max_new_tokens=256)

or pass it to llm():

llm(prompt, max_new_tokens=256)