bug: templating issue with mistral v0.3

janhq / cortex.tensorrt-llm

Cortex.Tensorrt-LLM is a C++ inference library that can be loaded by any server at runtime. It submodules NVIDIA’s TensorRT-LLM for GPU accelerated inference on NVIDIA's GPUs.

Apache License 2.0

37 stars 2 forks source link

~$ cortex models start mistral:7b-tensorrt-llm-linux-ampere { message: "Model loaded successfully", modelId: "mistral:7b-tensorrt-llm-linux-ampere", } ~$ cortex chat Inorder to exit, type 'exit()'. >> hi there Hello! How can I help you today? Feel free to ask me anything, from general knowledge questions to fun conversation topics. Is there something specific on your mind that you'd like to discuss or learn about? >> tell me about football Football is a popular sport played around the world with various versions and rules depending on the region. The most widely known version of football is Association Football (also called soccer), which was first codified in England during the mid-19th century by groups such as Sheffield F.C., the Cambridge Rules Conference, and the London FA Cup matches.In association football, teams aim to score goals by propelling a ball into their opponent’s goal using any part of their body except for arms and hands – except for the goalkeeper within his own penalty area. A standard match consists of two45-minute halves, making up90 minutes of play time plus added injury stoppages. If the game ends in a draw after regular time, it may proceed to extra time periods followed by penalties if necessary.The FIFA World Cup, held every four years since its inaugural tournament in Uruguay in1930, is considered the premier international competition in men's association football. There are also several other continental championship tournaments organized by confederations affiliated with FIFA: UEFA Champions League, Copa Libertadores, African Nations Championship, Asian Cup, CONCACAF Gold Cup, among others.Some countries have professional leagues where teams compete for domestic titles throughout the season, while others focus more on national team success at major competitions like the World Cup and Olympic Games. In North America, Major League Soccer (MLS) serves as the top tier league for both male and female players.Other forms of football include Rugby Union and Rugby League, Gaelic games, Australian Rules Football, and American football - each having distinct variations in rule sets but sharing similarities due to originating from early medieval folk games involving kicking a ball between opposing teams. >> thank you You're welcome! I'm always happy to help answer your questions. Let me know if you need assistance with anything else. Enjoy learning more about football! >> exit() Bye!

$ cortex run mistral:7b-tensorrt-llm-linux-ada
Inorder to exit, type 'exit()'.
>> hi there
Hello! How can I help you today? Feel free to ask me anything, from general knowledge questions to fun conversation topics. Is there something specific on 
your mind that you'd like to discuss or learn about?

>> how are you today?
As a model, I don't have feelings, but I am here and ready to assist with any question you may have! If it helps, let's say I feel as good as a well-oiled 
machine ;)Now, what would you like to talk about or learn more about today?

>> tell me a short story
Sure thing! Here is a short, lighthearted story for you:Once upon a time in the small town of Sunnyville, lived a young boy named Timmy who loved playing basketball more than anything else in the world. Every day after school, he could be found shooting hoops at his favorite spot behind the old library. One sunny afternoon while practicing his jump shot, an elderly woman approached him. She introduced herself as Mrs. Johnson and asked if she might watch him play since she had never seen anyone shoot so accurately before. Flattered by her compliment, Timmy happily agreed.As they chatted during breaks between shots, Mrs. Johnson shared stories about when she was younger, including tales of her own athletic prowess. It turned out this kindly senior citizen used to hold records for javelin throwing and long jumping in her prime! Intrigued, Timmy listened intently, feeling inspired by her strength and determination. He promised himself then and there that one day he too would make his mark in sports history.Over the next few months, their friendship grew stronger, and together they practiced every day until finally, competition season arrived. At first, things didn’t go according to plan; other players were faster, taller, and seemed to possess innate talents beyond compare. But instead of giving up hope, Timmy remembered the wisdom and encouragement he received from his new friend. With renewed vigor, he focused on improving each aspect of his game – agility, accuracy, speed, endurance - determined not just to win games but also to honor the legacy set forth by Mrs. Johnson.The big tournament eventually came around, and as luck would have it, both Timmy and Mrs. Johnson qualified 
for separate events within the same championship weekend. On Saturday morning, they competed side by side in track races; though she couldn't keep pace anymore due to age, seeing her cheering loudly spurred him onward towards victory lane. Later that evening, under bright stadium lights, Timmy stepped onto center court prepared to face off against some fierce opponents. The crowd roared as they watched him give everything he had, leaping high above defenders to score point after point. And in those final moments, when all seemed lost and the odds appeared insurmountable, he summoned the very lastounce of energy left inside him...and sank the winning basket!Exhausted yet exhilarated, Timmy embraced his fellow competitors, showering them with praise and gratitude for such an incredible contest. When the dust settled and trophies were distributed, he walked away proud knowing that despite initial failures, hard work and perseverance led him down a path filled with triumph and personal growth.But most importantly, throughout these trials and tribulations, Timmy cherished the bond formed over shared dreams and mutual respect. Forged through sweaty practices and heartfelt conversations, it became clear that true champions aren't defined solely by medals or titles. Instead, it lies deep within our hearts, where compassion, camaraderie, and unwavering spirit reside – qualities exemplified best by none other than Mrs. Johnson.

>> thank you
You're welcome! Let me know if you need anything else.

cortex run mistral:7b-tensorrt-llm-windows-ampere >> hi there Hello! How can I help you today? Feel free to ask me anything, from general knowledge questions to fun conversation topics. Is there something specific on your mind that you'd like to discuss or learn about? >> tell me something cool Sure thing! Did you know that the Great Barrier Reef is not just one long reef but actually a collection of over2900 individual reefs and900 islands? It stretches for more than1430 miles (over2300 kilometers) off the coast of Australia and is home to an incredible diversity of marine life including corals, fish, sharks, sea turtles, dugongs, and even whales. The reef system supports an incredibly rich ecosystem with around1500 species of fish and36 species of hard coral, making it one of the most biologically diverse places in the world. It's also the largest living structure on Earth, visible from space. Pretty amazing, right?! >> yes, it is amazing I'm glad you found it interesting! There are many other fascinating facts out there waiting to be discovered. If you have any other curiosities or would like to chat about different subjects, feel free to let me know. Enjoy learning new things! >> exit()

cortex run mistral:tensorrt-llm-windows-ada >> hi there Hello! How can I help you today? Feel free to ask me anything, from general knowledge questions to fun conversation topics. Is there something specific on your mind that you'd like to discuss or learn about? >> how are you today? As a model, I don't have feelings, but I am here and ready to assist with any question you may have! If it helps, let's say I feel as good as a well-oiled machine ;)Now, what would you like to talk about or learn more about today? >> tell me a short story Sure thing! Here is a short, lighthearted story for you:Once upon a time in the small town of Sunnyville, lived a young boy named Timmy who loved playing basketball more than anything else in the world. Every day after school, he could be found shooting hoops at his favorite spot behind the old library. One sunny afternoon while practicing his jump shot, an elderly woman approached him. She introduced herself as Mrs. Johnson and asked if she might watch him play since she had never seen anyone shoot so accurately before. Flattered by her compliment, Timmy happily agreed.As they chatted during breaks between shots, Mrs. Johnson shared stories about when she was younger, including tales of her own athletic prowess. It turned out this kindly senior citizen used to hold records for javelin throwing and long jumping in her prime! Intrigued, Timmy listened intently, feeling inspired by her strength and determination. He promised himself then and there that one day he too would make his mark in sports history.Over the next few months, their friendship grew stronger, and together they practiced every day until finally, competition season arrived. At first, things didn’t go according to plan; other players were faster, taller, and seemed to possess innate talents beyond compare. But instead of giving up hope, Timmy remembered the wisdom and encouragement he received from his new friend. With renewed vigor, he focused on improving each aspect of his game – agility, accuracy, speed, endurance - determined not just to win games but also to honor the legacy set forth by Mrs. Johnson.The big tournament eventually came around, and as luck would have it, both Timmy and Mrs. Johnson qualified for separate events within the same championship weekend. On Saturday morning, they competed side by side in track races; though she couldn't keep pace anymore due to age, seeing her cheering loudly spurred him onward towards victory lane. Later that evening, under bright stadium lights, Timmy stepped onto center court prepared to face off against some fierce opponents. The crowd roared as they watched him give everything he had, leaping high above defenders to score point after point. And in those final moments, when all seemed lost and the odds appeared insurmountable, he summoned the very lastounce of energy left inside him...and sank the winning basket!Exhausted yet exhilarated, Timmy embraced his fellow competitors, showering them with praise and gratitude for such an incredible contest. When the dust settled and trophies were distributed, he walked away proud knowing that despite initial failures, hard work and perseverance led him down a path filled with triumph and personal growth.But most importantly, throughout these trials and tribulations, Timmy cherished the bond formed over shared dreams and mutual respect. Forged through sweaty practices and heartfelt conversations, it proved far more valuable than any medal ever could. From that moment forward, whenever someone told him "You Can't," he simply smiled and said, "Watch Me." >> thank you You're welcome! Let me know if you need assistance with anything else. Have a great rest of your day!

janhq / cortex.tensorrt-llm

bug: templating issue with mistral v0.3 #50