Open Harri1703 opened 11 months ago
If you are loading directly from Huggingface:
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained("TheBloke/zephyr-7B-beta-GGUF", gpu_layers=50)
llm.config.context_length = 8192
you can use this code to increase maximun context length for your llm config = {'max_new_tokens': 256, 'repetition_penalty': 1.1,'context_length':1000}
llm = CTransformers(model='marella/gpt-2-ggml', config=config) for more infromation you can check below links
https://python.langchain.com/docs/integrations/providers/ctransformers
https://github.com/marella/ctransformers#config