[Question]: Structured output not working for RAG application

[kushagr96]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

I am trying to use Structured output for my RAG application, but getting ValueError: unmatched '{' in format spec error.

Here's the code for generating structured output:

class Output(BaseModel):
    """Output containing the response, page numbers, and confidence."""
    response: str = Field(..., description="The answer to the question in less than 100 tokens.")
    document_id: List[int] = Field(
        ...,
        description="The id of the document used to answer this question. Do not include it if the context is irrelevant."
    )
    confidence: float = Field(
        ...,
        description="Confidence value between 0-1 of the correctness of the result."
    )
    confidence_explanation: str = Field(
        ..., description="Explanation for the confidence score"
    )

structured_llm= llm.as_structured_llm(output_cls=Output)
query_engine = doc_index.as_query_engine(
        similarity_top_k=4,
        llm=llm,
        response_mode="refine",
    )
response = query_engine.query("Who is the CEO of apple?")
print(str(response))

This generates the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_256/3541374038.py in <cell line: 2>()
      1 qe = get_query_engine(sllm_70b)
----> 2 response = qe.query("Who is the CEO of apple?")
      3 print(str(response))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py in wrapper(func, instance, args, kwargs)
    263             )
    264             try:
--> 265                 result = func(*args, **kwargs)
    266             except BaseException as e:
    267                 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/base/base_query_engine.py in query(self, str_or_query_bundle)
     50             if isinstance(str_or_query_bundle, str):
     51                 str_or_query_bundle = QueryBundle(str_or_query_bundle)
---> 52             query_result = self._query(str_or_query_bundle)
     53         dispatcher.event(
     54             QueryEndEvent(query=str_or_query_bundle, response=query_result)

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py in wrapper(func, instance, args, kwargs)
    263             )
    264             try:
--> 265                 result = func(*args, **kwargs)
    266             except BaseException as e:
    267                 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/query_engine/retriever_query_engine.py in _query(self, query_bundle)
    174         ) as query_event:
    175             nodes = self.retrieve(query_bundle)
--> 176             response = self._response_synthesizer.synthesize(
    177                 query=query_bundle,
    178                 nodes=nodes,

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py in wrapper(func, instance, args, kwargs)
    263             )
    264             try:
--> 265                 result = func(*args, **kwargs)
    266             except BaseException as e:
    267                 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/response_synthesizers/base.py in synthesize(self, query, nodes, additional_source_nodes, **response_kwargs)
    239             payload={EventPayload.QUERY_STR: query.query_str},
    240         ) as event:
--> 241             response_str = self.get_response(
    242                 query_str=query.query_str,
    243                 text_chunks=[

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py in wrapper(func, instance, args, kwargs)
    263             )
    264             try:
--> 265                 result = func(*args, **kwargs)
    266             except BaseException as e:
    267                 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/response_synthesizers/refine.py in get_response(self, query_str, text_chunks, prev_response, **response_kwargs)
    175                 # if this is the first chunk, and text chunk already
    176                 # is an answer, then return it
--> 177                 response = self._give_response_single(
    178                     query_str, text_chunk, **response_kwargs
    179                 )

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/response_synthesizers/refine.py in _give_response_single(self, query_str, text_chunk, **response_kwargs)
    232                     structured_response = cast(
    233                         StructuredRefineResponse,
--> 234                         program(
    235                             context_str=cur_text_chunk,
    236                             **response_kwargs,

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py in wrapper(func, instance, args, kwargs)
    263             )
    264             try:
--> 265                 result = func(*args, **kwargs)
    266             except BaseException as e:
    267                 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/response_synthesizers/refine.py in __call__(self, *args, **kwds)
     82             answer = answer.model_dump_json()
     83         else:
---> 84             answer = self._llm.predict(
     85                 self._prompt,
     86                 **kwds,

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py in wrapper(func, instance, args, kwargs)
    263             )
    264             try:
--> 265                 result = func(*args, **kwargs)
    266             except BaseException as e:
    267                 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/llms/llm.py in predict(self, prompt, **prompt_args)
    580         else:
    581             formatted_prompt = self._get_prompt(prompt, **prompt_args)
--> 582             response = self.complete(formatted_prompt, formatted=True)
    583             output = response.text
    584         parsed_output = self._parse_output(output)

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py in wrapper(func, instance, args, kwargs)
    263             )
    264             try:
--> 265                 result = func(*args, **kwargs)
    266             except BaseException as e:
    267                 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/llms/callbacks.py in wrapped_llm_predict(_self, *args, **kwargs)
    429                 )
    430                 try:
--> 431                     f_return_val = f(_self, *args, **kwargs)
    432                 except BaseException as e:
    433                     callback_manager.on_event_end(

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/llms/structured_llm.py in complete(self, prompt, formatted, **kwargs)
    139     ) -> CompletionResponse:
    140         complete_fn = chat_to_completion_decorator(self.chat)
--> 141         return complete_fn(prompt, **kwargs)
    142 
    143     @llm_completion_callback()

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/base/llms/generic_utils.py in wrapper(prompt, **kwargs)
    171         # normalize input
    172         messages = prompt_to_messages(prompt)
--> 173         chat_response = func(messages, **kwargs)
    174         # normalize output
    175         return chat_response_to_completion_response(chat_response)

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py in wrapper(func, instance, args, kwargs)
    263             )
    264             try:
--> 265                 result = func(*args, **kwargs)
    266             except BaseException as e:
    267                 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/llms/callbacks.py in wrapped_llm_chat(_self, messages, **kwargs)
    171                 )
    172                 try:
--> 173                     f_return_val = f(_self, messages, **kwargs)
    174                 except BaseException as e:
    175                     callback_manager.on_event_end(

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/llms/structured_llm.py in chat(self, messages, **kwargs)
    105         # the messages don't technically have any variables (they are already formatted)
    106 
--> 107         chat_prompt = ChatPromptTemplate(message_templates=_escape_json(messages))
    108 
    109         output = self.llm.structured_predict(

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/prompts/base.py in __init__(self, message_templates, prompt_type, output_parser, metadata, template_var_mappings, function_mappings, **kwargs)
    247         template_vars = []
    248         for message_template in message_templates:
--> 249             template_vars.extend(get_template_vars(message_template.content or ""))
    250 
    251         super().__init__(

/layers/local_env/requirements/lib/python3.10/site-packages/llama_index/core/prompts/utils.py in get_template_vars(template_str)
     10     formatter = Formatter()
     11 
---> 12     for _, variable_name, _, _ in formatter.parse(template_str):
     13         if variable_name:
     14             variables.append(variable_name)

ValueError: unmatched '{' in format spec

llama-index versions:

llama-cloud==0.0.17
llama-index==0.11.7
llama-index-agent-openai==0.3.1
llama-index-cli==0.3.0
llama-index-core==0.11.7
llama-index-embeddings-huggingface==0.3.1
llama-index-embeddings-openai==0.2.4
llama-index-indices-managed-llama-cloud==0.3.0
llama-index-legacy==0.9.48.post3
llama-index-llms-openai==0.2.3
llama-index-llms-openai-like==0.2.0
llama-index-multi-modal-llms-openai==0.2.0
llama-index-program-openai==0.2.0
llama-index-question-gen-openai==0.2.0
llama-index-readers-file==0.2.1
llama-index-readers-json==0.2.0
llama-index-readers-llama-parse==0.3.0
llama-parse==0.5.2

[dosubot[bot]]

@kushagr96 Hello! I'm a bot here to help you with bug fixes, answer your questions, and guide you on becoming a contributor. If you need assistance, feel free to ask!

To resolve the ValueError: unmatched '{' in format spec error in your RAG application, ensure that any curly brackets in the format instructions are properly escaped. The provided code already includes a mechanism to escape curly brackets in the format method of the LangchainOutputParser class. Here is the relevant part of the code:

def format(self, query: str) -> str:
    """Format a query with structured output formatting instructions."""
    format_instructions = self._output_parser.get_format_instructions()

    # TODO: this is a temporary hack. if there's curly brackets in the format
    # instructions (and query is a string template), we need to
    # escape the curly brackets in the format instructions to preserve the
    # overall template.
    query_tmpl_vars = {
        v for _, v, _, _ in Formatter().parse(query) if v is not None
    }
    if len(query_tmpl_vars) > 0:
        format_instructions = format_instructions.replace("{", "{{")
        format_instructions = format_instructions.replace("}", "}}")

    if self._format_key is not None:
        fmt_query = query.format(**{self._format_key: format_instructions})
    else:
        fmt_query = query + "\n\n" + format_instructions

    return fmt_query

This code checks if there are any template variables in the query string and escapes the curly brackets in the format_instructions to prevent the ValueError. Ensure that this logic is correctly implemented and that the format method is being used properly in your application. The LangchainOutputParser class is designed to handle this specific issue, so make sure you are utilizing this class and its format method as shown ^[1][2].

To continue talking to Dosu, mention @dosu.

[kushagr96]

Thanks, that worked!