[Bug]: ExtendedGenerationConfig complains about missing `stream`

[Manouchehri]

What happened?

I'm not sure why this only cropped up recently. Didn't run into it during the testing of #2964.

Relevant log output

openai.InternalServerError: Error code: 500 - {'error': {'message': "VertexAIException - async_completion.<locals>.ExtendedGenerationConfig.__init__() got an unexpected keyword argument 'stream'\nmodel: vertex_ai/gemini-1.5-pro-preview-0409\nmodel_group: gemini-1.5-pro-preview-0409\ndeployment: vertex_ai/gemini-1.5-pro-preview-0409\nvertex_project: litellm-epic\nvertex_location: asia-east1\n", 'type': None, 'param': None, 'code': 500}}

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

[Manouchehri]

Oh interesting, I only run into this if I manually set stream=False. When stream=True, there's no issue.

[krrishdholakia]

hey @Manouchehri this should be fixed in v1.35.20

[Manouchehri]

I'm still running into it with gemini-1.5-pro-preview-0409 on v1.35.21. Can you share the test you ran? Here's mine:

#!/usr/bin/env python3.11
# -*- coding: utf-8 -*-
# Author: David Manouchehri

import os
import asyncio
import openai
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

c_handler = logging.StreamHandler()
logger.addHandler(c_handler)

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_API_BASE = os.getenv("OPENAI_API_BASE") or "https://api.openai.com/v1"

client = openai.AsyncOpenAI(
    api_key=OPENAI_API_KEY,
    base_url=OPENAI_API_BASE,
)

async def main():
    response = await client.chat.completions.create(
        model="gemini-1.5-pro-preview-0409",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What’s in this image now?"
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                        }
                    }
                ]
            }
        ],
        stream=False,
        temperature=0.0,
    )

    logger.debug(response.model_dump_json(indent=2))

if __name__ == "__main__":
    asyncio.run(main())

[Manouchehri]

openai.InternalServerError: Error code: 500 - {'error': {'message': 'VertexAIException - Unknown field for GenerationConfig: stream\nmodel: vertex_ai/gemini-1.5-pro-preview-0409\nmodel_group: gemini-1.5-pro-preview-0409\ndeployment: vertex_ai/gemini-1.5-pro-preview-0409\nvertex_project: litellm-epic\nvertex_location: us-east4\n', 'type': None, 'param': None, 'code': 500}}

[krrishdholakia]

there's a couple of them in here - https://github.com/BerriAI/litellm/blob/26e9ae38ce92c2485709eaee789624704e70aa83/litellm/tests/test_amazing_vertex_completion.py#L639

[Manouchehri]

I don't see any stream=False tests in that file?

[Manouchehri]

This seems to be resolved. Not sure which commit fixed it.