Model output ends inappropriately as soon as pattern can be matched.

[JamesKCS]

Describe the issue as clearly as possible:

Edit: See my comment below, it is a simpler and perhaps more glaring example of the bug. In the comment below that, we see that the problem is not limited to regex.

When using regex, with a pattern such as .* at the end, the model's outputs seem to get cut off soon after the pattern is matched (even if it should continue outputing). Apologies in advance if this is not an Outlines issue and/or I am making a trivial error.

Steps/code to reproduce the bug:

import outlines

model = outlines.models.transformers("Mistral-7B-Instruct-v0.3-newer-from-8-16-24")

prompt = """[INST] Please tell me a story about a cat. [/INST]

"""

regex_str = r"Once upon a time, there was a cat named Snowball.*"

num_trials = 1
for _ in range(num_trials):
    generator = outlines.generate.regex(model, regex_str)
    prompts = [prompt, prompt, prompt, prompt]
    answers = generator(prompts)
    for answer in answers:
        print(answer)

Expected result:

I would expect it to generate a short story. But it seems to generate a minimal number of tokens and then get "cut off". Example outputs:

Once upon a time, there was a cat named Snowball living
Once upon a time, there was a cat named Snowball
Once upon a time, there was a cat named Snowball
Once upon a time, there was a cat named Snowball in

Note that when forcing a specific length, for example, .{100}, all works as expected.

Outlines/Python version information:

0.0.46 Python 3.10.14

[JamesKCS]

Another example of what I believe may be the same issue:

import outlines

model = outlines.models.transformers("Mistral-7B-Instruct-v0.3-newer-from-8-16-24")

prompt = "11 - 1 = "

regex_str = r"(10|[0-9])"

generator = outlines.generate.regex(model, regex_str)
prompts = [prompt, prompt, prompt, prompt]
answers = generator(prompts)
for answer in answers:
    print(answer)

It consistently outputs 1 instead of 10. My guess is that it is trying to output 10, but that the first token, 1, fulfills the regex pattern, so then it ends. When I change the regex pattern to r"(10|[2-9])", it correctly outputs 10, which supports this theory.

[JamesKCS]

Note that this bug is not limited to regex, it affects generate.choice too:

import outlines

model = outlines.models.transformers("Mistral-7B-Instruct-v0.3-newer-from-8-16-24")

prompt = "11 - 1 = "

choices = ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10"] # outputs 1
choices = ["0", "2", "3", "4", "5", "6", "7", "8", "9", "10"] # when 1 is removed, it outputs 10

generator = outlines.generate.choice(model, choices)
prompts = [prompt, prompt, prompt, prompt]
answers = generator(prompts)
for answer in answers:
    print(answer)

[JamesKCS]

I verified that this is not limited to the Mistral model above. I observed the same problem with Phi-3.5-mini-instruct.

[lapp0]

1) Please try Outlines pre-release: pip install git+https://github.com/outlines-dev/outlines --upgrade

2) If that doesn't work, please share the output of pip freeze, as I cannot reproduce:

~/outlines$ python3 repro_1121.py                                                                                                                                                                                                                                                                          
The `load_in_4bit` and `load_in_8bit` arguments are deprecated and will be removed in the future versions. Please, pass a `BitsAndBytesConfig` object in `quantization_config` argument instead.
Loading checkpoint shards: 100%|_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________| 3/3 [00:07<00:00,  2.50s/it]
Once upon a time, there was a cat named Snowball who lived in a small, snowy village at the northernmost tip of the world. Snowball was no ordinary cat. He had blue fur that glowed like the northern lights, and his eyes sparkled like the stars up above. Snowball's family had passed down the extraordinary blue fur for generations, and it was said that he could see things that other cats could not see. The village people loved Snowball and considered him a special friend. But little did they know, Snowball had a secret. He could hear the whispers of the wind and felt the pulse of the earth beneath his paws. He knew that deep in the cold, dark forest, a great danger was growing. A terrible storm was brewing, and if it reached the village, it would destroy everything. The villagers were blissfully unaware, enjoying the festive winter season, but Snowball knew the time had come for him to act. He decided to venture into the forest to warn them. The journey was treacherous, but Snowball's keen senses guided him forward, and he was able to make his way through the dense woods. Snowball approached the heart of the forest, and there he discoverd the source of the danger - a terrible ice villain named Frostbreach. Snowball witnessed as Frostbreach sought to harness the power of the winter storm to crush his beloved village. With courage in his heart and determination in his eyes, Snowball bravely challenged Frostbreach. The battle was fierce, but Snowball held his ground, using his extraordinary abilities to fend off Frostbreach's icy attacks. In the midst of the battle, Snowball remembered the love and warmth of his village, and the power of friendship that bound them all. With a final burst of energy, he managed to defeat Frostbreach and, with a sigh of relief, Snowball fragmented the storm before it could reach the village. The people were amazed to find their brave friend had saved them once again. From that day on, the villagers vowed to always stand together and show the world the power of love and friendship. And as for Snowball, he returned to his beloved village, forever their guardian and friend. His blue fur continued to glow, a reminder of the incredible things that can happen when we care for each other and stand up for what is right. The end.
Once upon a time, there was a cat named Snowball who lived in a bustling city. Unlike most cats, Snowball didn't enjoy lounging in the sun or chasing after laser pointers. Instead, he had a deep passion for helping others, especially the city's homeless population. He may have only been a cat, but he had a heart as big as any human's. He lived with the kind-hearted family of Bakers, who ran a local bakery that offered free meals to anyone in need. Snowball often accompanied Mrs. Baker on her daily trips to distribute food to the homeless. He was a source of comfort to those who couldn't admit to being lonely. His soft purrs and gentle presence brought smiles to faces, even in the hardest of times. One cold winter evening, while they were out distributing food, Snowball noticed a man huddled in an alleyway, trying to keep warm under a tattered blanket. The man's eyes sparkled with unshed tears as he stared at the warm meal Snowball's family was offering. Snowball felt a deep urge to help. He jumped out of Mrs. Baker's arms and scuttled over to the man. He nudged the man's blanket with his paw and meowed softly. The man hesitated at first, unsure of what to make of this friendly feline. But as Snowball began to groom himself. slowly, the man's guard came down, and he cautiously reached out to pet Snowball. The cat purred in response, and the man smiled for the first time in weeks. Snowball spent the rest of the night with the man, keeping him company and offering comfort. The following day, Mr. Baker discovered that the man had a job interview and found him work, ensuring he would never have to go back to sleeping in the cold alleys. Snowball's selfless act that cold winter night had a ripple effect, leading to a positive change in this man's life. And from then on, Snowball became known as the miraculous cat of the city, helping those in need, one purr at a time. His story served as a reminder to all that even the smallest of creatures can make a significant difference in the world.
Once upon a time, there was a cat named Snowball who lived in the picturesque village of Whispering Meadows. Snowball was a beautiful, fluffy Maine Coon with emerald green eyes that twinkled like the stars in the night sky. In the winter, her fur turned a stunning white, earning her the nickname "The Snow Queen." While Snowball was known for her delicate appearance, she was as tough as they come, with a heart of gold and a willingness to help others. Her entire demeanor radiated kindness and a deeply ingrained sense of compassion.  Despite her aloof exterior, she had a special place in the hearts of everyone in Whispering Meadows.  This was not just because of her striking appearance but because of her exceptional talents and her uncanny ability to help others, especially children. You see, Snowball had a knack for communicating with humans on a unique level. She could sense their emotions and respond to them in ways that no one in the village could ever imagine. Not only could she understand humans, but she could speak a few simple words that always seemed to be exactly what was needed to offer comfort and guidance. Snowball didn't just entertain the children with her antics or warm their laps at night; she guided and nurtured them, offering sage advice and a listening ear when they needed it the most. One winter night, as the winds howled and the snow blanketed the village, a little girl named Lucy found herself lost in the dark woods. Heartbroken and confused, Lucy wandered further from her home, certain that she would freeze to death in the cold. Just when all hope seemed lost, she heard a distinctive purr coming from the shadows. With a flick of her ear, there stood Snowball, beckoning Lucy towards her. Nonetheless, Lucy shivered and felt reluctant to approach the unknown while facing such a long and formidable journey. Snowball sensed her fear and paused for a moment before gently meowing, "Do not be afraid, dear child." Lucy cautiously took a step towards Snowball, and as she did, she felt a distinct warmth spread from her toes to the tips of her fingers, and her fear began to dissipate. With Snowball leading the way, Lucy was guided out of the woods and back to her home where her relieved parents awaited her. That night, Lucy's family knew that Snowball was more than just a cat. She was a friend, a guardian, and a guiding light in their lives. The villagers of Whispering Meadows continued to sing Snowball's praises, and her legend grew over the years -- a testament to the power of kindness, love, and the incredible bond between Snowball and the children of her dramatic, snow-covered village.
Once upon a time, there was a cat named Snowball, who lived in a quaint little village nestled between dense forests and towering mountains. Snowball was not your ordinary house cat; she was a sleek, chocolate-brown Siamese with glistening green eyes and a tail that resembled a furry whip. She was known throughout the village for her stealthy agility, mystifying intelligence, and enchanting purr. She could climb trees with ease, sneak into apartments undetected, and you could always find her nestled comfortably on a warm lap or sunlit windowsill. But what the villagers admired most was Snowball's uncanny ability to help lost souls find their way. When somebody was feeling lonely or disheartened, Snowball seemed to sense it and would offer comfort with her calming presence. Could it be her magic, or was it her understanding of human emotions that drew people to her? The villagers would often ponder about it while they admired the gentle feline. _ _ _ _ _ _ _ _ _ _ _ __ _ _ One day, as dusk set upon the village, a stranger arrived. His clothes were torn, his face rugged, and he looked dazed as he wandered aimlessly down the cobbled streets. The villagers, who had never seen the man before, decided he must have lost his way. Some chatted nervously among themselves while some called out to him; nevertheless, the stranger seemed unresponsive and continued to visibly struggle._ _ _ _ _ _ _ _ _ _ _ Sensing the human's distress, Snowball sprang into action. She crept over to the stranger and nudged his hand with hers. The man gradually looked down and meekly stroked the cat's soft fur, a slight smile beginning to form on his lips. As if magnetized, Snowball led the man towards the village square and to a cozy little inn, unbeknownst to her, it was what the man had been desperately looking for. As he entered the inn, weary and shivering from the cold, the villagers warmly welcomed him, offering him food, shelter, and companionship. From that day forth, the stranger was welcomed into the village with open arms. Some whispered that it was ultimately Snowball who had led him there, helping him find his way when all seemed lost. Snowball's legend only grew following this remarkable encounter, and villagers knew they were blessed to have her in their lives. From then on, they called her the Cat of the Mystical Pathfinders, for it was through her that lost souls found solace, and the weary discovered home once more.

[JamesKCS]

Thank you for your help. I do not have direct control over the packages and versions in this environment, and cannot easily add directly with pip (I did add the latest non-dev version of Outlines, but other packages might be older). I also am unable to share the full output of pip freeze.

Based on your response, it's likely that something that Outlines depends on is outdated. I'll mark this as resolved, but if someone else has this problem in the future, they should reopen.