Model Repeats Same Phrase

[sashokbg]

Hello, I am not sure if this is an issue related to lmql decoder ot the underlying model itself.

I am running as a server an instance of Mistral (llama.cpp format) with the following params:

[Loading llama.cpp model from llama.cpp:/home/alexander/Games2/lmql/models/mistral-7b-v0.1.Q5_K_M.gguf with {'n_ctx': 4096, 'n_gpu_layers': 35, 'repeat_penalty': 1.2, 'temp': 0.8, 'device_map': 'auto'} ]

My client in the playground has the following lmql code taken from the examples in the docs:

argmax
    # review to be analyzed
    review = """We had a great stay. Hiking in the mountains was fabulous and the food is really good."""

    # use prompt statements to pass information to the model
    "Review: {review}"
    "Q: What is the underlying sentiment of this review and why?"
    # template variables like [ANALYSIS] are used to generate text
    "A:[ANALYSIS]" where not "\n" in ANALYSIS

    # use constrained variable to produce a classification
    "Based on this, the overall sentiment of the message can be considered to be[CLS]"
from 
    lmql.model("llama.cpp:/home/alexander/Games2/lmql/models/mistral-7b-v0.1.Q5_K_M.gguf",
    endpoint="localhost:9999",
    tokenizer="mistralai/Mistral-7B-v0.1")
distribution
   CLS in [" positive", " neutral", " negative"]

Then the model keeps repeating over and over the same thing.

Review: We had a great stay. Hiking in the mountains was fabulous and the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food

Is this related to the argmax function and "where not "\n"" part ?

Thank you for your help. Best Regards Aleks

[sashokbg]

I am sorry I read more carefully the constraints part of the documentation and I found my answer:

https://lmql.ai/docs/language/constraints.html

A good combination of STOPS_AT and len(TOKENS) does the job like in the example:

"A story about life:[STORY]" \
   where STOPS_AT(STORY, ".") and len(TOKENS(STORY)) > 40