deepset-ai / haystack-core-integrations

Additional packages (components, document stores and the likes) to extend the capabilities of Haystack version 2.0 and onwards
https://haystack.deepset.ai
Apache License 2.0
100 stars 96 forks source link

Gemini integration does not work with `AnswerBuilder` #974

Open medsriha opened 1 month ago

medsriha commented 1 month ago

Describe the bug

I encountered the following error when connecting GoogleAIGeminiGenerator with the AnswerBuilder component.

haystack.core.errors.PipelineConnectError: Cannot connect 'llm.replies' with 'answer_builder.replies': their declared input and output types do not match.
'llm':
 - replies: List[Union[str, Dict[str, str]]]
'answer_builder':
 - replies: Union[List[str], List[ChatMessage]] (available)

To Reproduce

    from haystack import Pipeline
    from haystack.components.builders import AnswerBuilder
    from haystack_integrations.components.generators.google_ai import GoogleAIGeminiGenerator
    from haystack.utils import Secret

    gemini = GoogleAIGeminiGenerator(model="gemini-pro", api_key=Secret.from_env_var("GEMINI-API-KEY") )

    pipeline = Pipeline()

    pipeline.add_component(name="llm", instance=gemini)
    pipeline.add_component(name="answer_builder", instance=AnswerBuilder())

    pipeline.connect("llm.replies", "answer_builder.replies")

    res = pipeline.run(data={"llm": {"parts": "Hello"}, "answer_builder": {"query": "hello"}})

Describe your environment (please complete the following information):

lbux commented 3 weeks ago

This has to do with the way that the Google Generators are handling function calling (this also affects vertex). Other generators simply return a string with the message and the meta (which might contain the function data).

If the output type is changed to List[str], the error will go away, but we lose out on the function call return data. It can be added to meta if we also add meta as an output type.

I think there are three options:

acompa commented 6 days ago

Checking in as another affected party here. Thanks for investigating!