[Bug]: ollama_chat `max_tokens` (and possibly other params) ignored

[pchalasani]

What happened?

Litellm 1.19.4

The max_tokens param is ignored. Looking in the code, it looks like the get_optional_params fn doesn't pick it up, even though max_tokens is passed. Examples below, with ollama/llama2, ollama/mistral, ollama_chat/mistral.

Relevant log output

In [1]: from litellm import completion

In [2]: completion(model="ollama_chat/mistral", messages=[{"content":"why did world war2 end?", "role":"user"}], max_tokens=10)
Out[2]: ModelResponse(id='chatcmpl-efae6a1f-55f8-43f0-984e-7e3bf6e63ce6', choices=[Choices(finish_reason='stop', index=0, message={'role': 'assistant', 'content': " World War II ended due to a combination of military victories, political negotiations, and the exhaustion of resources among the major powers involved in the conflict. Here are some key events that contributed to the end of the war:\n\n1. Battle of Stalingrad (1942-1943): This was one of the deadliest battles in history, marking a turning point for the Soviet Union against Nazi Germany. The Soviet forces managed to stop the German advance and eventually pushed them back, causing heavy losses and demoralization within the German army.\n\n2. D-Day invasion (June 6, 1944): The Allied invasion of Normandy, France, was a significant turning point in Europe's favor. It allowed the Allies to establish a large foothold on the European continent and put pressure on Germany from multiple directions.\n\n3. Battle of Berlin (April-May 1945): In the final stages of the war, Soviet forces managed to encircle and capture the German capital, Berlin. Hitler committed suicide in his bunker on April 30, 1945, and the Nazi regime collapsed shortly thereafter.\n\n4. Unconditional surrender: The Allies demanded unconditional surrender from Germany, meaning that they would not negotiate any terms or grant concessions before accepting the defeat of Nazi Germany. This stance left little room for compromise and ultimately led to the end of the war in Europe.\n\n5. Atomic bombs: In August 1945, the United States dropped atomic bombs on Hiroshima and Nagasaki, Japan. These devastating attacks resulted in over 200,000 deaths and forced Japan to surrender unconditionally. The war in the Pacific ended on September 2, 1945.\n\nThe end of World War II marked a significant turning point in history, leading to the establishment of the United Nations and setting the stage for the Cold War between the Soviet Union and the Western Allies."})], created=1706273760, model='ollama/mistral', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=16, completion_tokens=430, total_tokens=446), _response_ms=12141.161)

In [3]: completion(model="ollama/mistral", messages=[{"content":"why did world war2 end?", "role":"user"}], max_tokens=10)
Out[3]: ModelResponse(id='chatcmpl-c942ddfc-8f88-4fd1-bf14-9a99baf222bf', choices=[Choices(finish_reason='stop', index=0, message=Message(content=" World War II ended with the unconditional surrender of the Axis powers. The major turning point came in 1944 and 1945 with a series of major military defeats for Germany and Japan.\n\nIn Europe, the Soviet Army's victory at the Battle of Stalingrad in January 1943 was a turning point. Then, in June 1944, the Allied invasion of Normandy marked the beginning of the end of the war in Europe. The Allies pushed into Germany from the west, while the Soviets advanced from the east. Berlin fell to the Soviets in May 1945. On May 7, 1945, Germany unconditionally surrendered, marking the end of the European War.\n\nIn the Pacific, the United States dropped atomic bombs on the Japanese cities of Hiroshima and Nagasaki in August 1945. The bombings devastated the cities and killed tens of thousands of people. Japan's leaders eventually surrendered unconditionally on September 2, 1945, bringing an end to the war in the Pacific.\n\nThe formal surrender ceremonies for both Europe and the Pacific took place on September 7, 1945, onboard the USS Missouri in Tokyo Bay. This day is now commemorated as Victory over Japan Day or V-J Day.", role='assistant'))], created=1706273816, model='ollama/mistral', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=7, completion_tokens=300, total_tokens=307), _response_ms=5169.532)

In [4]: completion(model="ollama/llama2", messages=[{"content":"why did world war2 end?", "role":"user"}], max_tokens=10)
Out[4]: ModelResponse(id='chatcmpl-049faa1b-fec5-4965-b3b4-4f00ef6794be', choices=[Choices(finish_reason='stop', index=0, message=Message(content="World War II ended in 1945 for a number of reasons, including:\n\n1. The defeat of Nazi Germany: In April 1945, Allied forces launched a massive offensive against Berlin, the capital of Nazi Germany. After weeks of intense fighting, Berlin fell to the Allies on May 2, 1945, and Hitler committed suicide on April 30, 1945. With their leader gone, the German government surrendered on May 7, 1945.\n2. The defeat of Japan: After the atomic bombings of Hiroshima and Nagasaki in August 1945, Japan realized that they could no longer continue fighting. On September 2, 1945, the Japanese government formally surrendered to the Allies on board the USS Missouri in Tokyo Bay.\n3. The Soviet Union's entry into the war: In June 1941, Germany launched Operation Barbarossa, a massive invasion of the Soviet Union. The Soviet counterattack was highly successful, and by December 1941, the Germans were forced to retreat. The Soviet Union continued to fight against the Nazis until their defeat in 1945.\n4. Economic exhaustion: Both the Axis powers (Germany, Italy, and Japan) and the Allies (United States, Great Britain, France, and the Soviet Union) were exhausted from years of fighting. The war had taken a huge toll on both sides, with millions of casualties and immense destruction.\n5. International pressure: The international community, including organizations like the United Nations, had been putting pressure on both sides to negotiate a peace settlement. The Soviet Union's entry into the war in 1941 and the atomic bombings of Hiroshima and Nagasaki in 1945 significantly shifted the balance of power in favor of the Allies.\n6. The rise of new technologies: During World War II, new technologies such as radar, sonar, jet engines, and nuclear weapons were developed. These technologies gave the Allies a significant advantage over the Axis powers, particularly in the Pacific Theater.\n7. Changes in leadership: As the war progressed, there were changes in leadership among the Axis powers. For example, Benito Mussolini was overthrown in Italy in 1943, and Hirohito's government in Japan became increasingly isolated as the war dragged on.\n8. Increased public opposition: As the war continued, opposition to the war effort grew both at home and abroad. This led to increased pressure on governments to end the conflict.\n\nThese factors combined to ultimately lead to the defeat of the Axis powers and the end of World War II.", role='assistant'))], created=1706273846, model='ollama/llama2', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=27, completion_tokens=611, total_tokens=638), _response_ms=14806.514)

Twitter / LinkedIn details

No response

[pchalasani]

I guess this is related to #1539

[krrishdholakia]

@pchalasani just pushed the fix - https://github.com/BerriAI/litellm/commit/37de964da47362729842f58baf228ae74314f7ab

[krrishdholakia]

Should be live soon in v1.20.6. Please re-open the issue if that doesn't fix your problem.