[Question]: Does PandasQueryEngine support multiple DataFrames?How to make it possible

[Chakradhar448]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

Hello, I am using "from llama_index.core.query_engine import PandasQueryEngine" to build "query_engine = PandasQueryEngine(df=df, verbose=True)".

What if i have a list of data Frames say "dfs = [df1,df2] and that the df1 and df2 can be merged together with a common column.

How do we generate the pandas expression for a given query in this case?

[dosubot[bot]]

To work with multiple DataFrames in PandasQueryEngine, you'll first need to merge these DataFrames into a single DataFrame. This is because PandasQueryEngine is designed to operate on a single DataFrame. Here's a step-by-step guide to handle your scenario:

1. Merge the DataFrames: Merge your DataFrames (df1, df2) on the common column. Use pd.merge(df1, df2, on='common_column') to create a single DataFrame that combines both df1 and df2.

2. Initialize PandasQueryEngine: Once you have your merged DataFrame, initialize the PandasQueryEngine with it. For example, query_engine = PandasQueryEngine(df=merged_df, verbose=True).

3. Prepare and Execute Your Query: With the engine initialized, you can now prepare your query. If you're looking to generate a pandas expression for a given query, ensure your query is well-defined and relevant to the merged DataFrame's structure.

4. Extract and Use the Generated Pandas Expression: After executing the query, you'll receive a response that includes the generated Pandas expression. This expression is what you'll use to perform further data manipulation or analysis.

Here's a simplified code snippet to illustrate these steps:

import pandas as pd
from llama_index.core.query_engine import PandasQueryEngine

# Example DataFrames
df1 = pd.DataFrame(...)  # Your DataFrame 1
df2 = pd.DataFrame(...)  # Your DataFrame 2

# Merge DataFrames
merged_df = pd.merge(df1, df2, on='common_column')

# Initialize the Query Engine
query_engine = PandasQueryEngine(df=merged_df, verbose=True)

# Example Query (replace with your actual query)
query_str = "your natural language query here"

# Execute the query (assuming you have a method to do so)
response = query_engine.query_method(query_str)

# Use the generated Pandas expression
pandas_expression = response  # This will be your Pandas code to execute

This approach ensures that you can work with multiple DataFrames by first merging them into a single DataFrame, which then allows you to utilize the PandasQueryEngine effectively for generating pandas expressions based on your queries.

Sources

• Querying with multiple sheets at once using Pandas query engine
• llama-index-legacy/llama_index/legacy/query_engine/pandas/pandas_query_engine.py
• llama-index-core/llama_index/core/schema.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.