Number of tokens (815) exceeded maximum context length (512).

[Gautam-Kantesariya]

I'm getting this error continuously. Which Parameters affect this?

[JWBWork]

Did you ever find a solution to this? I'm facing the same issue. I think this might be related to ctransformers/transformers in general than this specific model.

I was able to extend the context length somewhat which allows the conversation to continue a little longer before I exceed the context length. Once I exceed the context length the model just starts spewing a bunch of gibberish.

model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/WizardLM-7B-uncensored-GGML", 
    model_type="llama",
    context_length=2048,
)

I was at least able to silence these messages like so

import logging
logger = logging.getLogger("ctransformers")
logger.setLevel(logging.ERROR)

[Gautam-Kantesariya]

instead of CTransformers, I use llama cpp. for load the model.

model_path = hf_hub_download(
  repo_id="TheBloke/Llama-2-7b-Chat-GGUF",
  filename="llama-2-7b-chat.Q4_K_M.gguf",
  resume_download=True,
  cache_dir="/models",  #custom path for save the model
)

llm = LlamaCpp(
  model_path = model_path,
  n_gpu_layers=100, #According to your GPU if you have
  n_batch=2048,
  verbose=True,
  f16_kv=True,
  n_ctx=4096
)

[Kunjesh07]

instead of CTransformers, I use llama cpp. for load the model.

model_path = hf_hub_download(
  repo_id="TheBloke/Llama-2-7b-Chat-GGUF",
  filename="llama-2-7b-chat.Q4_K_M.gguf",
  resume_download=True,
  cache_dir="/models",  #custom path for save the model
)

llm = LlamaCpp(
  model_path = model_path,
  n_gpu_layers=100, #According to your GPU if you have
  n_batch=2048,
  verbose=True,
  f16_kv=True,
  n_ctx=4096
)

does it resolve using this ?

[manjunathshiva]

Yes because he increased the context length from default 512 to 4096. If you do not have GPU do not set n_gpu_layers