BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
13.88k stars 1.64k forks source link

[Bug]: Litellm proxy chat streaming from cohere with n>1 does not emit 'index' properly for the streams (always 0) #5236

Closed the-crypt-keeper closed 2 months ago

the-crypt-keeper commented 3 months ago

What happened?

Fire up litellm proxy aimed at any cohere model, for example litellm --port 4000 --model cohere/c4ai-aya-23-35b

Do an n=4 call with stream=false, observe that 'index' is returned correctly:

curl http://localhost:4000/v1/chat/completions -d '{ "n": 4, "messages": [{"role":"user","content":"Once upon a time,"}]}'

Now do an n=4 call with stream=true, observe that 'index' is always 0:

curl http://localhost:4000/v1/chat/completions -d '{ "n": 4, "messages": [{"role":"user","content":"Once upon a time,"}], "stream": true}'

Relevant log output

Without streaming, all is well:

{
  "id": "chatcmpl-39a0d841-59ae-4e1a-8d88-c0c6d60bc8d5",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 1,
      "message": {
        "content": "in a kingdom far away, there lived a brave and noble knight named Sir Landon. Sir Landon was known throughout the land for his skill with the sword and his unwavering dedication to protecting the innocent. He had a loyal steed named Silver, who was as swift as the wind and as brave as his master.\n\nSir Landon had heard tales of a mysterious dragon that lived in the distant mountains, a dragon that possessed immense power and guarded a treasure beyond imagination. Determined to face this mighty creature and secure the treasure for his kingdom, Sir Landon set out on a perilous journey, with Silver by his side.\n\nAs they ventured deeper into the mountains, they encountered treacherous paths and dangerous creatures. They fought off wolves and dodged the cunning eyes of eagles, their feathers glinting in the sunlight. The cold winds howled, but Sir Landon's heart remained steadfast.\n\nFinally, they reached the dragon's lair, a cavernous den hidden deep within the mountains. The air crackled with anticipation as the dragon, a massive creature with scales as shiny as armor and eyes that glowed with an otherworldly light, emerged.\n\n\"Who dares disturb my solitude?\" the dragon boomed, its voice echoing through the cavern.\n\nWith a steady hand and a brave heart, Sir Landon stepped forward. \"I am Sir Landon, a knight of this realm, and I have come to face you and claim the treasure you guard.\"\n\nThe dragon laughed, a sound that shook the very foundations of the mountain. \"None have ever dared to challenge me, mortal knight. But I shall grant you a chance to prove your worth. Three trials you must pass, and only then shall you face me in battle.\"\n\nSir Landon's heart raced, but his resolve did not waver. He accepted the dragon's challenge and embarked on a series of trials that tested his strength, cunning, and bravery. He fought off wild beasts, solved ancient riddles, and navigated treacherous mazes. With each trial, he grew stronger, his skills honed to perfection.\n\nFinally, the day of the final battle arrived. Sir Landon, with Silver by his side, faced the dragon once more. The air crackled with magic as their blades clashed, sparks flying with each blow. It was a fierce and brutal battle, but Sir Landon's skill and determination proved too much for the dragon.\n\nWith one final, mighty blow, Sir Landon landed a critical strike, and the dragon fell. The treasure, a gleaming chest filled with gold and jewels, lay at Sir Landon's feet.\n\nAs the dragon's breath faded, it spoke with a voice now calm and serene. \"You have proven your worth, Sir Landon. Take the treasure, and may it bring prosperity to your kingdom. Remember, true valor lies not only in strength but also in compassion and honor.\"\n\nSir Landon, with a heart full of gratitude and a sense of accomplishment, thanked the dragon. He returned to his kingdom, where he was greeted as a hero. The treasure he brought back helped the kingdom flourish, and Sir Landon's name was forever etched in the annals of its history.\n\nAnd so, the tale of Sir Landon, the brave knight who faced the mysterious dragon, lived on, inspiring future generations with its message of courage, determination, and honor.",
        "role": "assistant"
      }
    },
    {
      "finish_reason": "stop",
      "index": 2,
      "message": {
        "content": "in a kingdom far away, there lived a brave and noble prince. He was known for his kindness and courage, and he had a special connection with the animals of the kingdom.\n\nThe prince spent his days exploring the enchanted forests, climbing the majestic mountains, and swimming in the crystal-clear lakes. He formed strong bonds with the creatures he encountered, from the wise old owls to the playful foxes.\n\nOne day, while wandering through the woods, the prince stumbled upon a mysterious cave. As he entered, he discovered a magical world filled with sparkling gems and ancient treasures. But he soon realized that the cave was not empty—it was home to a wise and powerful dragon.\n\nThe dragon, with its shimmering scales and piercing eyes, revealed to the prince that it was guarding a secret that would change the kingdom forever. It spoke of an ancient prophecy, foretelling the arrival of a hero who would bring peace and harmony to the land.\n\nIntrigued and honored, the prince set out on a quest to fulfill the prophecy. He faced many challenges and overcame obstacles, always guided by his courage and the support of his animal friends. He battled evil sorcerers, rescued fair maidens, and brought justice to the kingdom.\n\nAs the prince's reputation grew, so did the hope and joy in the hearts of the people. They saw him as a symbol of unity and strength, a true champion of their realm. The prince, with his loyal companions by his side, continued his journey, always striving to make the kingdom a better place.\n\nIn the end, the prince fulfilled the prophecy and brought about a new era of peace and prosperity. The dragon, grateful for his bravery and dedication, granted him a special blessing. The prince was now forever linked to the magical world he had discovered, and his name would be remembered for generations to come.\n\nAnd so, the tale of the brave prince and his enchanted kingdom was passed down through the ages, inspiring courage and kindness in the hearts of all who heard it.",
        "role": "assistant"
      }
    },
    {
      "finish_reason": "stop",
      "index": 3,
      "message": {
        "content": "in a kingdom far away, there lived a brave and noble prince. He was known for his kindness and courage, and he had a strong desire to make a positive impact on the world. One day, while exploring the kingdom, the prince stumbled upon a mysterious old castle. As he entered, he discovered a magical library filled with ancient tomes and scrolls. \n\nAmong the many books, he found a magical map that was said to lead to a hidden treasure. The map showed a path through a treacherous mountain range, and it was guarded by a wise and powerful dragon. Undeterred by the challenge, the prince set out on his quest, determined to retrieve the treasure and use its power for the betterment of his kingdom. \n\nAs he ventured through the mountains, he encountered many obstacles and dangers, but his bravery and resourcefulness guided him forward. He faced wild beasts, navigated treacherous terrain, and solved ancient puzzles set by the dragon to protect the map. Finally, he reached the dragon's lair, and a great battle ensued. The prince's courage and skill prevailed, and he emerged victorious. \n\nThe dragon, impressed by the prince's determination and noble cause, granted him the treasure he sought. It was a magical gem that held the power to grant any wish. The prince returned to his kingdom, and with the gem's power, he brought peace and prosperity to his people. He used the treasure wisely, ensuring that everyone had access to abundance and happiness. \n\nThe kingdom flourished under the prince's rule, and his name became legendary. His tale was told through the ages, inspiring future generations to embark on their own quests for knowledge, treasure, and the betterment of the world. And so, the brave prince lived on in the hearts and minds of all who heard his story, a reminder that courage, kindness, and a noble cause can overcome any obstacle. \n\nAnd so, the tale of the brave prince and his magical quest concludes, leaving a legacy of inspiration and a reminder that within each of us lies the power to make a positive impact on the world.",
        "role": "assistant"
      }
    },
    {
      "finish_reason": "stop",
      "index": 4,
      "message": {
        "content": "in a kingdom far away, there lived a brave and noble knight named Sir Landon. Sir Landon was known throughout the land for his skill in battle and his unwavering dedication to protecting the innocent. He had a loyal steed named Silver, who was as swift as the wind and as brave as his master.\n\nSir Landon had heard tales of a powerful dragon that had been terrorizing the nearby villages, and he was determined to put an end to its reign of terror. He set out on his quest, his heart filled with courage and his mind focused on the task at hand.\n\nAs he ventured into the treacherous mountains where the dragon made its lair, Sir Landon encountered many challenges. He fought off wild beasts, navigated through treacherous terrain, and braved the harsh elements. But his determination never wavered.\n\nFinally, he reached the dragon's lair, a cavernous den deep within the heart of the mountain. The air was thick with the smell of sulfur, and the ground was littered with the remains of the dragon's previous victims.\n\nWithout hesitation, Sir Landon drew his sword and charged towards the dragon. The beast was massive, its scales shimmering with an otherworldly light, and its eyes glowing with an intense heat. It breathed fire, and the very air around it seemed to crackle with power.\n\nThe knight and the dragon engaged in a fierce battle, their clashes echoing throughout the mountains. Sir Landon's sword clashed against the dragon's scales, and he dodged the beast's fiery breath with lightning-quick reflexes. It was a battle of strength, skill, and willpower.\n\nJust when it seemed that the dragon might overcome the knight, Sir Landon's loyal steed, Silver, charged into the fray. Together, they fought as a team, their movements synchronized and their attacks coordinated. The dragon, despite its mighty power, could not match their speed and agility.\n\nFinally, with one swift and decisive blow, Sir Landon plunged his sword into the dragon's heart. The beast roared in pain and then fell silent, its massive body collapsing to the ground. The cave shook with the impact, and the air filled with a sense of calm.\n\nThe villagers, who had been praying for their hero's success, cheered as the news of the dragon's defeat spread. Sir Landon had saved the kingdom from the terror of the dragon, and his name would forever be remembered as a champion of justice and bravery.\n\nFrom that day forward, Sir Landon and Silver continued to embark on new adventures, always ready to face any challenge that threatened the peace and happiness of the kingdom they loved so dearly. Their tale became a legend, inspiring future generations of knights to follow in their courageous footsteps.\n\nAnd so, the story of Sir Landon and the Dragon continues to be told, a reminder of the power of courage, determination, and the bond between a knight and his loyal steed. May their legacy live on forever in the hearts of all who hear their tale.",
        "role": "assistant"
      }
    }
  ],
  "created": 1723768076,
  "model": "c4ai-aya-23-35b",
  "object": "chat.completion",
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 689,
    "total_tokens": 694
  }
}

but with streaming, 'index' doesnt work it's always 0

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":"in","role":"assistant"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":"in"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" a"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":"in"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" a"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":"in"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" kingdom"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" a"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" kingdom"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" a"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" far"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" kingdom"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" far"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" kingdom"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" far"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" away"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" away"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" far"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":","}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" away"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":","}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":" away"}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-102e47c7-dab2-4425-b469-cbd52c67f99a","choices":[{"index":0,"delta":{"content":","}}],"created":1723768105,"model":"c4ai-aya-23-35b","object":"chat.completion.chunk"}


### Twitter / LinkedIn details

_No response_
krrishdholakia commented 2 months ago

this is now fixed - https://github.com/BerriAI/litellm/blob/8b23a2334f883370a0aa76c1c74b2bc5b56dffa4/litellm/utils.py#L10253