Issue Deploying Pipeline with LLMs in It

Hello,

I'm encountering an issue when trying to deploy a RAG (Retrieval-Augmented Generation) application using Hayhooks and the Haystack framework. I have two pipelines: postgres_indexing and postgres_query. The postgres_indexing pipeline deploys without any problems, but when I attempt to deploy the postgres_query pipeline and then view the Swagger documentation, I receive the following error:

pydantic.errors.pydanticinvalidforjsonschema: Cannot generate a JsonSchema for core_schema.CallableSchema

Detailed Error Traceback:

INFO:     192.168.65.1:21879 - "GET /openapi.json HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/opt/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/opt/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/opt/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/opt/venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/opt/venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/opt/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1009, in openapi
    return JSONResponse(self.openapi())
                        ^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/fastapi/applications.py", line 981, in openapi
    self.openapi_schema = get_openapi(
                          ^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/fastapi/openapi/utils.py", line 475, in get_openapi
    field_mapping, definitions = get_definitions(
                                 ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/fastapi/_compat.py", line 230, in get_definitions
    field_mapping, definitions = schema_generator.generate_definitions(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 379, in generate_definitions
    self.generate_inner(schema)
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 527, in new_handler_func
    json_schema = js_modify_function(schema_or_field, current_handler)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/main.py", line 694, in __get_pydantic_json_schema__
    return handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 527, in new_handler_func
    json_schema = js_modify_function(schema_or_field, current_handler)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 272, in modify_model_json_schema
    json_schema = handler(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1415, in model_schema
    json_schema = self.generate_inner(schema['schema'])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1510, in model_fields_schema
    json_schema = self._named_required_fields_schema(named_required_fields)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1318, in _named_required_fields_schema
    field_json_schema = self.generate_inner(field).copy()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 545, in new_handler_func
    json_schema = js_modify_function(schema_or_field, current_handler)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2469, in json_schema_update_func
    json_schema = {**handler(core_schema_or_field), **json_schema_update}
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1386, in model_field_schema
    return self.generate_inner(schema['schema'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 527, in new_handler_func
    json_schema = js_modify_function(schema_or_field, current_handler)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/main.py", line 694, in __get_pydantic_json_schema__
    return handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 527, in new_handler_func
    json_schema = js_modify_function(schema_or_field, current_handler)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 272, in modify_model_json_schema
    json_schema = handler(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1415, in model_schema
    json_schema = self.generate_inner(schema['schema'])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1510, in model_fields_schema
    json_schema = self._named_required_fields_schema(named_required_fields)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1318, in _named_required_fields_schema
    field_json_schema = self.generate_inner(field).copy()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 545, in new_handler_func
    json_schema = js_modify_function(schema_or_field, current_handler)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2469, in json_schema_update_func
    json_schema = {**handler(core_schema_or_field), **json_schema_update}
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1386, in model_field_schema
    return self.generate_inner(schema['schema'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1042, in default_schema
    json_schema = self.generate_inner(schema['schema'])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 1105, in nullable_schema
    inner_json_schema = self.generate_inner(schema['schema'])
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 552, in generate_inner
    json_schema = current_handler(schema)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 37, in __call__
    return self.handler(core_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 511, in handler_func
    json_schema = generate_for_schema_type(schema_or_field)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 835, in callable_schema
    return self.handle_invalid_for_json_schema(schema, 'core_schema.CallableSchema')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pydantic/json_schema.py", line 2185, in handle_invalid_for_json_schema
    raise PydanticInvalidForJsonSchema(f'Cannot generate a JsonSchema for {error_info}')
pydantic.errors.PydanticInvalidForJsonSchema: Cannot generate a JsonSchema for core_schema.CallableSchema

For further information visit https://errors.pydantic.dev/2.9/u/invalid-for-json-schema

Initially thought the issue was with the streaming_callback in the OpenAIGenerator component (llm), so I removed it entirely from both the code and the YAML configuration. For reference, here is my YAML file:

components:
  embedder:
    init_parameters:
      api_base_url: null
      api_key:
        env_vars:
        - OPENAI_API_KEY
        strict: true
        type: env_var
      dimensions: null
      model: text-embedding-3-small
      organization: null
      prefix: ''
      suffix: ''
    type: haystack.components.embedders.openai_text_embedder.OpenAITextEmbedder
  llm:
    init_parameters:
      api_base_url: null
      api_key:
        env_vars:
        - OPENAI_API_KEY
        strict: true
        type: env_var
      generation_kwargs: {}
      model: gpt-4o-mini
      organization: null
      # streaming_callback: null
      system_prompt: null
    type: haystack.components.generators.openai.OpenAIGenerator
  prompt_builder:
    init_parameters:
      required_variables: null
      template: "\n    Given these documents, answer the question.\n    Documents:\n\
        \    {% for doc in documents %}\n        {{ doc.content }}\n    {% endfor\
        \ %}\n    Question: {{question}}\n    Answer:\n    "
      variables: null
    type: haystack.components.builders.prompt_builder.PromptBuilder
  retriever:
    init_parameters:
      document_store:
        init_parameters:
          connection_string:
            env_vars:
            - PG_CONN_STR
            strict: true
            type: env_var
          embedding_dimension: 1536
          hnsw_ef_search: null
          hnsw_index_creation_kwargs: {}
          hnsw_index_name: haystack_hnsw_index
          hnsw_recreate_index_if_exists: false
          keyword_index_name: haystack_keyword_index
          language: english
          recreate_table: false
          search_strategy: hnsw
          table_name: sherlock-vectors
          vector_function: cosine_similarity
        type: haystack_integrations.document_stores.pgvector.document_store.PgvectorDocumentStore
      filter_policy: replace
      filters: {}
      top_k: 5
      vector_function: cosine_similarity
    type: haystack_integrations.components.retrievers.pgvector.embedding_retriever.PgvectorEmbeddingRetriever
connections:
- receiver: retriever.query_embedding
  sender: embedder.embedding
- receiver: prompt_builder.documents
  sender: retriever.documents
- receiver: llm.prompt
  sender: prompt_builder.prompt
max_loops_allowed: 100
metadata: {}

and here is my Python code that generated it:

 # Initializing components
    prompt_template = """
    Given these documents, answer the question.
    Documents:
    {% for doc in documents %}
        {{ doc.content }}
    {% endfor %}
    Question: {{question}}
    Answer:
    """
    embedder = OpenAITextEmbedder(model="text-embedding-3-small")
    retriever = PgvectorEmbeddingRetriever(document_store=doc_store, top_k=5)
    prompt_builder = PromptBuilder(template=prompt_template)
    llm = OpenAIGenerator(model="gpt-4o-mini",
                          api_key=Secret.from_env_var('OPENAI_API_KEY'),
                          streaming_callback=None)

    # Creating pipeline
    rag_pipeline = Pipeline()
    rag_pipeline.add_component("embedder", embedder)
    rag_pipeline.add_component("retriever", retriever)
    rag_pipeline.add_component("prompt_builder", prompt_builder)
    rag_pipeline.add_component("llm", llm)

    # Connecting components
    rag_pipeline.connect("embedder.embedding", "retriever.query_embedding")
    rag_pipeline.connect("retriever", "prompt_builder.documents")
    rag_pipeline.connect("prompt_builder", "llm")

    rag_pipeline.draw(path=Path("rag_pipeline.png"))

    print(rag_pipeline.dumps())

What is going on? Why am I having trouble with the querying pipeline? I've narrowed it down to the LLM; when removing this component from the pipeline the deployment goes smoothly. Appreciate the help in advance.

deepset-ai / hayhooks

Issue Deploying Pipeline with LLMs in It #38