[Bug]: Various Issues and Concerns Associated with Sub Question Query Engine

HarikrishnanK9 commented 1 month ago

Bug Description

I have tried different open source llms,as well as gpt 3.5 turbo;Except for gpt for all other llms i had various kinds of issues like coroutine,key error,etc (For majority of my questions i got appropriate response) Currently iam using microsoft phi3 mini 4k instruct as per your documentation page with the combination of subquestion query engine.I have tried distinctive names for documents to ensure query engine tools are not identical to avoid possible Hallucinations.But i felt coroutine error and key error for some question i have asked with sub question query engine,the error is as follows:

Traceback (most recent call last): File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/async_utils.py", line 33, in asyncio_run return loop.run_until_complete(coro) File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/async_utils.py", line 79, in _gather return await asyncio.gather(*tasks_to_execute) File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/query_engine/sub_question_query_engine.py", line 232, in _aquery_subq query_engine = self._query_engines[sub_q.tool_name] KeyError: 'LMAR'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/paperspace/clinsight/backend/dataingestion/test/h_dev/demo3_working.py", line 256, in result = qa_test(input_dict) File "/home/paperspace/clinsight/backend/dataingestion/test/h_dev/demo3_working.py", line 241, in qa_test qa_response = sub_query_engine.query(f"{query}") File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 230, in wrapper result = func(*args, *kwargs) File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/base/base_query_engine.py", line 52, in query query_result = self._query(str_or_query_bundle) File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 230, in wrapper result = func(args, **kwargs) File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/query_engine/sub_question_query_engine.py", line 156, in _query qa_pairs_all = run_async_tasks(tasks) File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/async_utils.py", line 81, in run_async_tasks outputs: List[Any] = asyncio_run(_gather()) File "/home/paperspace/anaconda3/envs/finetune_env/lib/python3.10/site-packages/llama_index/core/async_utils.py", line 47, in asyncio_run raise type(e)(f"Error running coroutine: {e!s}") from e KeyError: "Error running coroutine: 'LMAR'"

Another notable thing found that if OPENAI API KEY is in our .env fille and at the same time if we are using another open source llms like llama2 70b,zypher,etc the SubQuestionQueryEngine will utilize the openai key without our permission.Its a serious matter of concern as we are properly specified in variables which llm should be taken.

If an organization deals with confidential documents,it may cause data leakage and possible legal actions against them.

Version

llama-index==0.10.57

Steps to Reproduce

try Sub Question Query Engine[ https://docs.llamaindex.ai/en/stable/examples/query_engine/sub_question_query_engine/ ] with open source llms specified in [ https://docs.llamaindex.ai/en/stable/module_guides/models/llms/ ] remember input datas are in .pkl form (pdfs files are parsed into .pkl files );I have tried 2 dozen of different combinations with SubQuestionQueryEngine (phi3mini4k,zyphr,llama,gpt3.5,etc).unfortunately iam not able to share the code due to confidentiality.Yet why iam reporting this because the current bug can sue several startups and sometimes llamaindex due to dataleakage.

Relevant Logs/Tracbacks

No response

dosubot[bot] commented 1 month ago

Hey @HarikrishnanK9! 👋 I'm here to help you out with any bugs or questions you have. I can also help you become a contributor.

To address the issues you're encountering with the SubQuestionQueryEngine, follow these steps:

Resolve the KeyError: 'LMAR': Ensure that the QueryEngineTool with the metadata.name 'LMAR' is correctly added to the query_engine_tools list when initializing the SubQuestionQueryEngine. Here's an example:

from llama_index.core.tools.query_engine import QueryEngineTool
from llama_index.core.query_engine.sub_question_query_engine import SubQuestionQueryEngine
from llama_index.core.schema import ToolMetadata

# Assuming you have a query engine for 'LMAR'
lmar_query_engine = YourLMARQueryEngine()  # Replace with your actual query engine

# Create a QueryEngineTool for 'LMAR'
lmar_tool = QueryEngineTool(
   metadata=ToolMetadata(name='LMAR', description='Description of LMAR tool'),
   query_engine=lmar_query_engine
)

# Initialize the SubQuestionQueryEngine with the required tools
sub_query_engine = SubQuestionQueryEngine.from_defaults(
   query_engine_tools=[lmar_tool, ...]  # Add other tools as needed
)

# Now you can use sub_query_engine to query

Ensure that the metadata.name for each QueryEngineTool matches the tool_name used in your sub-questions ^[1].

Prevent Unintentional Use of OpenAI API Key: Modify the resolve_openai_credentials function to ensure it only uses the OpenAI API key when explicitly required. Add a condition to check if the OpenAI API key is needed before resolving it:

def resolve_openai_credentials(
   api_key: Optional[str] = None,
   api_base: Optional[str] = None,
   api_version: Optional[str] = None,
   use_openai: bool = False  # Add a flag to indicate if OpenAI should be used
) -> Tuple[Optional[str], str, str]:
   """Resolve OpenAI credentials.

   The order of precedence is:
   1. param
   2. env
   3. openai module
   4. default
   """
   if not use_openai:
       return None, "", ""  # Return empty credentials if OpenAI is not to be used

   # resolve from param or env
   api_key = get_from_param_or_env("api_key", api_key, "OPENAI_API_KEY", "")
   api_base = get_from_param_or_env("api_base", api_base, "OPENAI_API_BASE", "")
   api_version = get_from_param_or_env(
       "api_version", api_version, "OPENAI_API_VERSION", ""
   )

   # resolve from openai module or default
   final_api_key = api_key or openai.api_key or ""
   final_api_base = api_base or openai.base_url or DEFAULT_OPENAI_API_BASE
   final_api_version = api_version or openai.api_version or DEFAULT_OPENAI_API_VERSION

   return final_api_key, str(final_api_base), final_api_version

When calling resolve_openai_credentials, specify whether the OpenAI API key should be used:

api_key, api_base, api_version = resolve_openai_credentials(use_openai=True)

This modification ensures that the OpenAI API key is only resolved when explicitly required, preventing its unintentional use when working with other open-source LLMs ^[2].

Logging and Debugging: Utilize the logging and debugging tools available within the LlamaIndex framework to trace and diagnose issues. Enable debug logging in your application:

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

Additionally, use the LlamaDebugHandler to print the trace of the sub-questions captured by the SUB_QUESTION callback event type:

from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler

# Using the LlamaDebugHandler to print the trace of the sub questions
# captured by the SUB_QUESTION callback event type
llama_debug = LlamaDebugHandler(print_trace_on_end=True)
callback_manager = CallbackManager([llama_debug])

Settings.callback_manager = callback_manager

These tools will help you trace and diagnose issues by providing structured logging and event-based callbacks ^[3]^[4]^[5]^[6].

To continue talking to Dosu, mention @dosu.

HarikrishnanK9 commented 1 month ago

@dosu when somebody uses pip install llama-index-core it will automatically installs openai related dpendencies automatically.for example i have created a new environment named index_env and tried pip install llama-index-core,not only installed llama-index-core==0.10.60 but also openai==1.38.0 ;So when a user uses open source llms by keeping openai key in his .env had no direct idea about this because there is another installation recommended in your github page " pip install llama-index-llms-openai " to install openai related dependencies seperately!!. if i use pip install llama-index-core and separately deletes openai,after that when i installs pip install llama-index-llms-openai then also i get both the openai==1.38.0 and llama-index-llms-openai==0.1.27 ;so i request you to separate openai installation from pip install llama-index-core otherwise user can directly istall openai separately.Remember this is just a suggestion to overcome to avoid unintentional use or installation of openai that sometimes user may not aware it exists or not

I will inspect the responses you have shared associated with SubQuestionQueryEngine later ,will compare with my existing code anyway and will share if any conflicts arise.

But the issues associated with openai in this message is always a concern not only for SubQuestionQuery but also for any other

logan-markewich commented 1 month ago

The error you are getting here is the LLM hallucinating the name of a query engine. This is pretty common for most open-source llms, as they aren't totally reliable. Maybe try llama3.1 (my recommendation is using Ollama), as that seems to be the best right now.

Also fyi, llama-index is a starter package around several namespaces sub-packages. You can always install llama-index-core and whatever else you need.

There is a Settings object that controls global defaults. If you don't want to use openai, set these to something else.

run-llama / llama_index