[Bug]: SubQuery Engine parsing error

Bug Description

I am getting a parsing error when using the SubQuestionQueryEngine.

llama_index.output_parsers.base.OutputParserException: Got invalid JSON object. Error: Extra data: line 11 column 1 (char 247) while scanning for the next token found character '`' that cannot start any token in "", line 11, column 1:

    ^. Got JSON string: [
    {
        "sub_question": "What are the research methods used in Iran",
        "tool_name": "iran_research"
    },
    {
        "sub_question": "What are the research methods used in Poland",
        "tool_name": "poland_research"
    }
]

Version

0.8.21

Steps to Reproduce

function used to generate error:

def compare_contrast(doc1:Document, doc1name: str, doc1desc:str, doc2: Document, doc2name: str, doc2desc: str, query: str): """ desc: generates compare and contrast response

args: 
    doc1: a file upload from st.file_uploader()
    doc1name: a name for doc1
    doc1desc: a short description of doc1
    doc2: a file upload from st.file_uploader() 
    doc2name: a name for doc2
    docdesc2: a short description of doc2
    query: a user query

return: 
    a compare and contrast agent response based on user query
"""
#Azure embeddings
embedding_llm = LangchainEmbedding(
                OpenAIEmbeddings(
                    model="text-embedding-ada-002",
                    deployment="text-embedding-ada-002",
                ),
                embed_batch_size=1,
            )

embeddings = OpenAIEmbeddings(deployment="text-embedding-ada-002", 
                              engine="text-embedding-ada-002", 
                              chunk_size=1)

# Completion model
llm_completion = AzureOpenAI(
    model="gpt-35-turbo1",
    engine="text-davinci-003"
)

# Chat Model
llm_chat = AzureChatOpenAI(deployment_name="gpt-35-turbo1", 
                            temperature=0, 
                            openai_api_version="2023-03-15-preview") 

llm_completion = AzureOpenAI(deployment_name="gpt-35-turbo1", 
                                model_name="text-davinci-003")

#llm_predictor = LLMPredictor(llm=llm_completion)

# service context
service_context = ServiceContext.from_defaults(llm=llm_completion,
                                                embed_model=embedding_llm) 

# create documents
doc1 = create_document(doc1)
doc2 = create_document(doc2) 

# indexes for each documnet
doc1_index = VectorStoreIndex.from_documents(documents=doc1, service_context=service_context)
doc2_index = VectorStoreIndex.from_documents(documents=doc2, service_context=service_context)

# query engines for each document
doc1_engine = doc1_index.as_query_engine(similarity_top_k=5)
doc2_engine = doc2_index.as_query_engine(similarity_top_k=5)

# tools
query_engine_tools = [
    QueryEngineTool(
        query_engine=doc1_engine,
        metadata=ToolMetadata(
            name=doc1name,
            description=doc1desc,
        ),
    ),
    QueryEngineTool(
        query_engine=doc2_engine,
        metadata=ToolMetadata(
            name=doc2name,
            description=doc2desc,
        ),
    ),
]

# subquery engine
s_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools, 
                                                service_context=service_context)

# # response
response = s_engine.query(query)

return response.response

Relevant Logs/Tracbacks

No response

🤖

Hello,

Thank you for providing detailed information about the issue you're facing. It seems like you're encountering a parsing error with the SubQuestionQueryEngine in LlamaIndex version 0.8.21 when trying to generate a compare and contrast response.

Based on a similar issue that was previously resolved in the LlamaIndex repository (#6993), it appears that the error might be due to the question generator you're using. The LLMQuestionGenerator can sometimes lead to output parsing errors. To eliminate these errors, the GuidanceQuestionGenerator is recommended, which is powered by the GuidancePydanticProgram and guarantees the desired structured output.

Here's how you can use the GuidanceQuestionGenerator:

from llama_index.question_gen.guidance_generator import GuidanceQuestionGenerator
from guidance.llms import OpenAI as GuidanceOpenAI

question_gen = GuidanceQuestionGenerator.from_defaults(
    guidance_llm=GuidanceOpenAI("text-davinci-003"), verbose=False
)

And then use this question_gen when constructing your SubQuestionQueryEngine:

s_engine = SubQuestionQueryEngine.from_defaults(
    question_gen=question_gen,  # use guidance based question_gen defined above
    query_engine_tools=query_engine_tools,
)

This should help eliminate the parsing error you're experiencing. You can find more details in the guidance_sub_question.ipynb example notebook and the sub_question_query_engine.py source code.

Please try this solution and let me know if it resolves your issue or if you need further assistance.

Best, Dosu

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

🤖

Hello,

Thank you for providing detailed information about the issue you're facing. It seems like you're encountering a parsing error with the SubQuestionQueryEngine in LlamaIndex version 0.8.21 when trying to generate a compare and contrast response.

Based on a similar issue that was previously resolved in the LlamaIndex repository (#6993), it appears that the error might be due to the question generator you're using. The LLMQuestionGenerator can sometimes lead to output parsing errors. To eliminate these errors, the GuidanceQuestionGenerator is recommended, which is powered by the GuidancePydanticProgram and guarantees the desired structured output.

Here's how you can use the GuidanceQuestionGenerator:
from llama_index.question_gen.guidance_generator import GuidanceQuestionGenerator
from guidance.llms import OpenAI as GuidanceOpenAI

question_gen = GuidanceQuestionGenerator.from_defaults(
    guidance_llm=GuidanceOpenAI("text-davinci-003"), verbose=False
)
And then use this question_gen when constructing your SubQuestionQueryEngine:
s_engine = SubQuestionQueryEngine.from_defaults(
    question_gen=question_gen,  # use guidance based question_gen defined above
    query_engine_tools=query_engine_tools,
)
This should help eliminate the parsing error you're experiencing. You can find more details in the guidance_sub_question.ipynb example notebook and the sub_question_query_engine.py source code.

Please try this solution and let me know if it resolves your issue or if you need further assistance.

Best, Dosu

Sources

[Bug]: SubQuestionQueryEngine error

docs/examples/output_parsing/guidance_sub_question.ipynb

llama_index/query_engine/sub_question_query_engine.py

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

Will this work on Azure? i.e. from guidance.llms import OpenAI as GuidanceOpenAI

The solution is not working, I am getting the error: llama_index.output_parsers.base.OutputParserException: Failed to parse pydantic object from guidance program.

Is there an Azure version that needs to be used?

updated code below:

def compare_contrast(doc1:Document, doc1name: str, doc1desc:str, doc2: Document, doc2name: str, doc2desc: str, query: str): """ desc: generates compare and contrast response

args: 
    doc1: a file upload from st.file_uploader()
    doc1name: a name for doc1
    doc1desc: a short description of doc1
    doc2: a file upload from st.file_uploader() 
    doc2name: a name for doc2
    docdesc2: a short description of doc2
    query: a user query

return: 
    a compare and contrast agent response based on user query
"""
#Azure embeddings
embedding_llm = LangchainEmbedding(
                OpenAIEmbeddings(
                    model="text-embedding-ada-002",
                    deployment="text-embedding-ada-002",
                ),
                embed_batch_size=1,
            )

embeddings = OpenAIEmbeddings(deployment="text-embedding-ada-002", 
                              engine="text-embedding-ada-002", 
                              chunk_size=1)

# Completion model
llm_completion = AzureOpenAI(
    model="gpt-35-turbo1",
    engine="text-davinci-003"
)

# Chat Model
llm_chat = AzureChatOpenAI(deployment_name="gpt-35-turbo1", 
                            temperature=0, 
                            openai_api_version="2023-03-15-preview") 

llm_completion = AzureOpenAI(deployment_name="gpt-35-turbo1", 
                                model_name="text-davinci-003")

#llm_predictor = LLMPredictor(llm=llm_completion)

# service context
service_context = ServiceContext.from_defaults(llm=llm_completion,
                                                embed_model=embedding_llm) 

# create documents
doc1 = create_document(doc1)
doc2 = create_document(doc2) 

# indexes for each documnet
doc1_index = VectorStoreIndex.from_documents(documents=doc1, service_context=service_context)
doc2_index = VectorStoreIndex.from_documents(documents=doc2, service_context=service_context)

# query engines for each document
doc1_engine = doc1_index.as_query_engine(similarity_top_k=5)
doc2_engine = doc2_index.as_query_engine(similarity_top_k=5)

# tools
query_engine_tools = [
    QueryEngineTool(
        query_engine=doc1_engine,
        metadata=ToolMetadata(
            name=doc1name,
            description=doc1desc,
        ),
    ),
    QueryEngineTool(
        query_engine=doc2_engine,
        metadata=ToolMetadata(
            name=doc2name,
            description=doc2desc,
        ),
    ),
]

# guidance generator
question_gen = GuidanceQuestionGenerator.from_defaults(
    guidance_llm=GuidanceOpenAI("text-davinci-003"), verbose=False
)

# subquery engine
s_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools, 
                                                service_context=service_context,
                                                question_gen=question_gen)

# response
response = s_engine.query(query)

return response.response

Switching to gpt35-turbo solved the issue, no need for guidance.

run-llama / llama_index