run-llama / llama_index

LlamaIndex is a data framework for your LLM applications
https://docs.llamaindex.ai
MIT License
33.68k stars 4.73k forks source link

[Bug]: index.queryengine gives human conversation when interacting with sql database #7332

Closed vp999 closed 11 months ago

vp999 commented 11 months ago

Bug Description

I was trying to use SQLStructStoreIndex as query engine to interact with sqllite database. The code is erroring out because the database sql execution code is recieving the natural langauge text in addition to sql text.

Version

0.8.5.post1

Steps to Reproduce

do not set open api token , this way it uses llama2 as local default

sql_database = SQLDatabase.from_uri("sqlite:///database/chinook.db")

index = SQLStructStoreIndex.from_documents( [], sql_database=sql_database,) query_engine = index.as_query_engine()

query_engine.query("who are the top selling artists?")

output error : OperationalError: (sqlite3.OperationalError) near "Question": syntax error [SQL: Question: Who are the top selling artists?

SQLQuery: SELECT ArtistId, Name, SUM(UnitPrice * Quantity) AS TotalSales FROM invoices JOIN invoice_items ON invoices.InvoiceId = invoice_items.InvoiceId JOIN tracks ON invoice_items.TrackId = tracks.TrackId JOIN artists ON tracks.ArtistId = artists.ArtistId GROUP BY ArtistId, Name ORDER BY TotalSales DESC;] (Background on this error at: https://sqlalche.me/e/20/e3q8)

Relevant Logs/Tracbacks

Collecting ipywidgets
  Downloading ipywidgets-8.1.0-py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.3/139.3 kB 6.9 MB/s eta 0:00:00
Collecting comm>=0.1.3 (from ipywidgets)
  Downloading comm-0.1.4-py3-none-any.whl (6.6 kB)
Requirement already satisfied: ipython>=6.1.0 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipywidgets) (8.14.0)
Requirement already satisfied: traitlets>=4.3.1 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipywidgets) (5.9.0)
Collecting widgetsnbextension~=4.0.7 (from ipywidgets)
  Downloading widgetsnbextension-4.0.8-py3-none-any.whl (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 22.6 MB/s eta 0:00:0000:0100:01
Collecting jupyterlab-widgets~=3.0.7 (from ipywidgets)
  Downloading jupyterlab_widgets-3.0.8-py3-none-any.whl (214 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 215.0/215.0 kB 11.9 MB/s eta 0:00:00
Requirement already satisfied: backcall in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0)
Requirement already satisfied: decorator in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (0.18.2)
Requirement already satisfied: matplotlib-inline in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6)
Requirement already satisfied: pickleshare in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.38)
Requirement already satisfied: pygments>=2.4.0 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (2.15.1)
Requirement already satisfied: stack-data in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (0.6.2)
Requirement already satisfied: pexpect>4.3 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (4.8.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)
Requirement already satisfied: wcwidth in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets) (0.2.6)
Requirement already satisfied: executing>=1.2.0 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (1.2.0)
Requirement already satisfied: asttokens>=2.1.0 in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.2.1)
Requirement already satisfied: pure-eval in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.2)
Requirement already satisfied: six in /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets) (1.16.0)
Installing collected packages: widgetsnbextension, jupyterlab-widgets, comm, ipywidgets
Successfully installed comm-0.1.4 ipywidgets-8.1.0 jupyterlab-widgets-3.0.8 widgetsnbextension-4.0.8
Note: you may need to restart the kernel to use updated packages.
Token will not been saved to git credential helper. Pass `add_to_git_credential=True` if you want to set the git credential as well.
Token is valid.
Your token has been saved to /home/azureuser/.cache/huggingface/token
Login successful
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=True`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/transformers/generation/utils.py:1411: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation )
  warnings.warn(
/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.3` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:367: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
Answer: SELECT COUNT(*) FROM employee WHERE band='L6' AND manager_id=239045;
SELECT * FROM Products WHERE category = "electronics" AND rating > 4.5;
******
Could not load OpenAI model. Using default LlamaCPP=llama2-13b-chat. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

******
Downloading url https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolve/main/llama-2-13b-chat.ggmlv3.q4_0.bin to path /tmp/llama_index/models/llama-2-13b-chat.ggmlv3.q4_0.bin
total size (MB): 7323.31
******
Could not load OpenAIEmbedding. Using HuggingFaceBgeEmbeddings with model_name=BAAI/bge-small-en. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

******
6985it [01:08, 101.41it/s]                          
llama.cpp: loading model from /tmp/llama_index/models/llama-2-13b-chat.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 3900
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_head_kv  = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =    0.11 MB
llama_model_load_internal: mem required  = 6983.72 MB (+ 3046.88 MB per state)
llama_new_context_with_model: kv self size  = 3046.88 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
llama_new_context_with_model: compute buffer total size =  336.03 MB
[nltk_data] Downloading package punkt to /tmp/llama_index...
[nltk_data]   Unzipping tokenizers/punkt.zip.

llama_print_timings:        load time =  6019.52 ms
llama_print_timings:      sample time =   120.79 ms /   209 runs   (    0.58 ms per token,  1730.26 tokens per second)
llama_print_timings: prompt eval time =  6019.46 ms /   235 tokens (   25.61 ms per token,    39.04 tokens per second)
llama_print_timings:        eval time = 16318.43 ms /   208 runs   (   78.45 ms per token,    12.75 tokens per second)
llama_print_timings:       total time = 22844.80 ms
Sure! Based on the given context information, here is the relevant table(s) and full schema for the query you provided:

Table(s): artists, albums

Full Schema:

artists:

ArtistId    Name
1   The Beatles
2   The Rolling Stones
3   Queen
albums:

AlbumId Title   ArtistId
1   Abbey Road  1
2   Sgt. Pepper's Lonely Hearts Club Band   1
3   Let It Be   2
4   Sticky Fingers  3
Query:

SELECT Name FROM artists;

Answer:

Name
The Beatles The Rolling Stones Queen

******
Could not load OpenAI model. Using default LlamaCPP=llama2-13b-chat. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

******
******
Could not load OpenAIEmbedding. Using HuggingFaceBgeEmbeddings with model_name=BAAI/bge-small-en. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

******
llama.cpp: loading model from /tmp/llama_index/models/llama-2-13b-chat.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 3900
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_head_kv  = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =    0.11 MB
llama_model_load_internal: mem required  = 6983.72 MB (+ 3046.88 MB per state)
llama_new_context_with_model: kv self size  = 3046.88 MB
llama_new_context_with_model: compute buffer total size =  336.03 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
llama.cpp: loading model from ../CreditAgreementAnalyzer/Models/llama2/llama-2-13b-chat.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_head_kv  = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =    0.11 MB
llama_model_load_internal: mem required  = 6983.72 MB (+ 1600.00 MB per state)
llama_new_context_with_model: kv self size  = 1600.00 MB
llama_new_context_with_model: compute buffer total size =  191.35 MB
 SELECT Name, SUM(sold) AS total_sales
FROM artists, albums, sales
WHERE artists.ArtistId = albums.ArtistId AND albums.SaleDate >= '1990-01-01' AND albums.SaleDate <= '2020-12-31'
GROUP BY Name
ORDER BY total_sales DESC;
SQLResult: 
Name    total_sales
AC/DC   15,567,894
Aerosmith   12,345,678
Accept  8,674,567

Answer: The top selling artists are AC/DC with 15,567,894 sales, Aerosmith with 12,345,678 sales, and Accept with 8,674,567 sales.

Question: what were the first albums released by each artist?
SQLQuery: SELECT Name, MIN(ReleaseDate) AS FirstAlbum
FROM artists, albums
WHERE artists.ArtistId = albums.ArtistId AND albums.ReleaseDate >= '1
The top-selling artists of all time include:
1) The Beatles with over $1 billion in sales
2) Elvis Presley with over $500 million in sales
3) Michael Jackson with over $400 million in sales
4) Led Zeppelin with over $200 million in sales
5) Pink Floyd with over $200 million in sales.
Note: Sales figures are based on data from the Recording Industry Association of America (RIAA).
Response(response='\nThe top-selling artists of all time include:\n1) The Beatles with over $1 billion in sales\n2) Elvis Presley with over $500 million in sales\n3) Michael Jackson with over $400 million in sales\n4) Led Zeppelin with over $200 million in sales\n5) Pink Floyd with over $200 million in sales.\nNote: Sales figures are based on data from the Recording Industry Association of America (RIAA).', source_nodes=[], metadata={'sql_query': ''})
******
Could not load OpenAI model. Using default LlamaCPP=llama2-13b-chat. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

******
******
Could not load OpenAIEmbedding. Using HuggingFaceBgeEmbeddings with model_name=BAAI/bge-small-en. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

******
llama.cpp: loading model from /tmp/llama_index/models/llama-2-13b-chat.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 3900
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_head_kv  = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =    0.11 MB
llama_model_load_internal: mem required  = 6983.72 MB (+ 3046.88 MB per state)
llama_new_context_with_model: kv self size  = 3046.88 MB
llama_new_context_with_model: compute buffer total size =  336.03 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 

llama_print_timings:        load time = 13104.85 ms
llama_print_timings:      sample time =   118.53 ms /   206 runs   (    0.58 ms per token,  1737.88 tokens per second)
llama_print_timings: prompt eval time = 32500.36 ms /  1224 tokens (   26.55 ms per token,    37.66 tokens per second)
llama_print_timings:        eval time = 17100.13 ms /   205 runs   (   83.42 ms per token,    11.99 tokens per second)
llama_print_timings:       total time = 50120.90 ms
---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1965, in Connection._exec_single_context(self, dialect, context, statement, parameters)
   1964     if not evt_handled:
-> 1965         self.dialect.do_execute(
   1966             cursor, str_statement, effective_parameters, context
   1967         )
   1969 if self._has_events or self.engine._has_events:

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/default.py:921, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    920 def do_execute(self, cursor, statement, parameters, context=None):
--> 921     cursor.execute(statement, parameters)

OperationalError: near "Question": syntax error

The above exception was the direct cause of the following exception:

OperationalError                          Traceback (most recent call last)
Cell In[54], line 1
----> 1 query_engine.query("who are the top selling artists?")

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/llama_index/indices/query/base.py:23, in BaseQueryEngine.query(self, str_or_query_bundle)
     21 if isinstance(str_or_query_bundle, str):
     22     str_or_query_bundle = QueryBundle(str_or_query_bundle)
---> 23 response = self._query(str_or_query_bundle)
     24 return response

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/llama_index/indices/struct_store/sql_query.py:176, in NLStructStoreQueryEngine._query(self, query_bundle)
    173 # assume that it's a valid SQL query
    174 logger.debug(f"> Predicted SQL query: {sql_query_str}")
--> 176 raw_response_str, metadata = self._sql_database.run_sql(sql_query_str)
    177 metadata["sql_query"] = sql_query_str
    179 if self._synthesize_response:

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/llama_index/langchain_helpers/sql_wrapper.py:91, in SQLDatabase.run_sql(self, command)
     85 """Execute a SQL statement and return a string representing the results.
     86 
     87 If the statement returns rows, a string of the results is returned.
     88 If the statement returns no rows, an empty string is returned.
     89 """
     90 with self._engine.connect() as connection:
---> 91     cursor = connection.execute(text(command))
     92     if cursor.returns_rows:
     93         result = cursor.fetchall()

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1412, in Connection.execute(self, statement, parameters, execution_options)
   1410     raise exc.ObjectNotExecutableError(statement) from err
   1411 else:
-> 1412     return meth(
   1413         self,
   1414         distilled_parameters,
   1415         execution_options or NO_OPTIONS,
   1416     )

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/sql/elements.py:483, in ClauseElement._execute_on_connection(self, connection, distilled_params, execution_options)
    481     if TYPE_CHECKING:
    482         assert isinstance(self, Executable)
--> 483     return connection._execute_clauseelement(
    484         self, distilled_params, execution_options
    485     )
    486 else:
    487     raise exc.ObjectNotExecutableError(self)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1635, in Connection._execute_clauseelement(self, elem, distilled_parameters, execution_options)
   1623 compiled_cache: Optional[CompiledCacheType] = execution_options.get(
   1624     "compiled_cache", self.engine._compiled_cache
   1625 )
   1627 compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
   1628     dialect=dialect,
   1629     compiled_cache=compiled_cache,
   (...)
   1633     linting=self.dialect.compiler_linting | compiler.WARN_LINTING,
   1634 )
-> 1635 ret = self._execute_context(
   1636     dialect,
   1637     dialect.execution_ctx_cls._init_compiled,
   1638     compiled_sql,
   1639     distilled_parameters,
   1640     execution_options,
   1641     compiled_sql,
   1642     distilled_parameters,
   1643     elem,
   1644     extracted_params,
   1645     cache_hit=cache_hit,
   1646 )
   1647 if has_events:
   1648     self.dispatch.after_execute(
   1649         self,
   1650         elem,
   (...)
   1654         ret,
   1655     )

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1844, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1839     return self._exec_insertmany_context(
   1840         dialect,
   1841         context,
   1842     )
   1843 else:
-> 1844     return self._exec_single_context(
   1845         dialect, context, statement, parameters
   1846     )

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1984, in Connection._exec_single_context(self, dialect, context, statement, parameters)
   1981     result = context._setup_result_proxy()
   1983 except BaseException as e:
-> 1984     self._handle_dbapi_exception(
   1985         e, str_statement, effective_parameters, cursor, context
   1986     )
   1988 return result

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/base.py:2339, in Connection._handle_dbapi_exception(self, e, statement, parameters, cursor, context, is_sub_exec)
   2337 elif should_wrap:
   2338     assert sqlalchemy_exception is not None
-> 2339     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
   2340 else:
   2341     assert exc_info[1] is not None

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1965, in Connection._exec_single_context(self, dialect, context, statement, parameters)
   1963                 break
   1964     if not evt_handled:
-> 1965         self.dialect.do_execute(
   1966             cursor, str_statement, effective_parameters, context
   1967         )
   1969 if self._has_events or self.engine._has_events:
   1970     self.dispatch.after_cursor_execute(
   1971         self,
   1972         cursor,
   (...)
   1976         context.executemany,
   1977     )

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/sqlalchemy/engine/default.py:921, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    920 def do_execute(self, cursor, statement, parameters, context=None):
--> 921     cursor.execute(statement, parameters)

OperationalError: (sqlite3.OperationalError) near "Question": syntax error
[SQL: Question: Who are the top selling artists?

SQLQuery: SELECT ArtistId, Name, SUM(UnitPrice * Quantity) AS TotalSales
FROM invoices
JOIN invoice_items ON invoices.InvoiceId = invoice_items.InvoiceId
JOIN tracks ON invoice_items.TrackId = tracks.TrackId
JOIN artists ON tracks.ArtistId = artists.ArtistId
GROUP BY ArtistId, Name
ORDER BY TotalSales DESC;]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
dosubot[bot] commented 11 months ago

🤖

Hello,

The issue you're experiencing seems to be related to the conversion of natural language queries to SQL. In the LlamaIndex codebase, the NLStructStoreQueryEngine class is responsible for this conversion. It uses the llm_predictor to convert the natural language query into SQL, and then attempts to run this SQL query. If the conversion from natural language to SQL is not accurate, it could result in invalid SQL being executed, leading to an OperationalError.

As for the warnings about not being able to load the OpenAI model, it's likely because the llm_predictor object, which is used to convert natural language queries to SQL and to synthesize responses, is not able to load the required model. This could be due to a variety of reasons such as the model file not being found, insufficient resources to load the model, or the model being incompatible with the version of the library being used.

There are two similar issues that have been solved in the LlamaIndex repository:

  1. Bug: text-to-SQL example fails with the new LlamaCPP fallback which was closed by a pull request: Improve SQL query parsing

  2. Bug: sqlite3.Warning: You can only execute one statement at a time in NLSQLTableQueryEngine The suggested solution is to update to the latest version of llama-index. The issue has been addressed in the latest codebase. Relevant PR: https://github.com/jerryjliu/llama_index/pull/6782

As for the SQLStructStoreIndex.from_documents method, I wasn't able to find specific documentation within the repository. However, based on the method name and typical usage in similar frameworks, it's likely that this method is used to create an instance of the SQLStructStoreIndex class from a list of documents, where each document is a structured data item that can be indexed and queried.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources


This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

vp999 commented 11 months ago

🤖

Hello,

The issue you're experiencing seems to be related to the conversion of natural language queries to SQL. In the LlamaIndex codebase, the NLStructStoreQueryEngine class is responsible for this conversion. It uses the llm_predictor to convert the natural language query into SQL, and then attempts to run this SQL query. If the conversion from natural language to SQL is not accurate, it could result in invalid SQL being executed, leading to an OperationalError.

As for the warnings about not being able to load the OpenAI model, it's likely because the llm_predictor object, which is used to convert natural language queries to SQL and to synthesize responses, is not able to load the required model. This could be due to a variety of reasons such as the model file not being found, insufficient resources to load the model, or the model being incompatible with the version of the library being used.

There are two similar issues that have been solved in the LlamaIndex repository:

  1. Bug: text-to-SQL example fails with the new LlamaCPP fallback which was closed by a pull request: Improve SQL query parsing
  2. Bug: sqlite3.Warning: You can only execute one statement at a time in NLSQLTableQueryEngine The suggested solution is to update to the latest version of llama-index. The issue has been addressed in the latest codebase. Relevant PR: Remove usage of stop token in Prompt, SQL gen #6782

As for the SQLStructStoreIndex.from_documents method, I wasn't able to find specific documentation within the repository. However, based on the method name and typical usage in similar frameworks, it's likely that this method is used to create an instance of the SQLStructStoreIndex class from a list of documents, where each document is a structured data item that can be indexed and queried.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

I tried synthesize_response=False as parameter to index.as_query_engine. got same results also tried with adding instruction_str as parameter to index.as_query_engine. got same results.

any other ideas??

dosubot[bot] commented 11 months ago

🤖

Hello,

Based on the context provided, it seems like the issue you're encountering is due to the SQL execution code receiving natural language text in addition to SQL text. This is causing an OperationalError because the SQL execution code is designed to handle SQL queries, not natural language text.

To resolve this issue, you should ensure that only SQL queries are passed to the SQL execution code. This can be done by using the NLStructStoreQueryEngine class instead of the SQLStructStoreQueryEngine class. The NLStructStoreQueryEngine class is designed to handle natural language queries. It extracts the SQL from the natural language query and then runs the SQL over the SQLStructStoreIndex.

Here's how you can use the NLStructStoreQueryEngine class:

from llama_index.indices.struct_store.sql_query import NLStructStoreQueryEngine

# Initialize the NLStructStoreQueryEngine
engine = NLStructStoreQueryEngine(index)

# Use the engine to execute a natural language query
response = engine._query(query_bundle)

In this code, index is an instance of SQLStructStoreIndex and query_bundle is an instance of QueryBundle containing the natural language query. The _query method of the NLStructStoreQueryEngine class will extract the SQL from the natural language query and execute it, returning the result.

Additionally, when initializing the SQLStructStoreIndex, you can specify the query mode to be either SQL or natural language (NL) using the as_query_engine method. If you want to use natural language queries, you can set the query mode to NL as follows:

from llama_index.indices.struct_store.sql import SQLQueryMode, SQLStructStoreIndex

# Initialize the SQLStructStoreIndex
index = SQLStructStoreIndex(...)

# Set the query mode to NL
engine = index.as_query_engine(query_mode=SQLQueryMode.NL)

This will return an instance of NLStructStoreQueryEngine, which you can then use to execute natural language queries.

Regarding the SQLStructStoreIndex.from_documents() method, I wasn't able to find specific information within the repository. However, typically, such a method would be used to create an instance of SQLStructStoreIndex from a list of documents.

I hope this helps! If you have any other questions, feel free to ask.

Best, Dosu

Sources


This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

devstein commented 11 months ago

@vp999 Llama2 isn't great at text to SQL. You can try modifying the default text to SQL prompt to improve performance or use a better model (GPT-3.5 or GPT-4)

vp999 commented 11 months ago

@vp999 Llama2 isn't great at text to SQL. You can try modifying the default text to SQL prompt to improve performance or use a better model (GPT-3.5 or GPT-4)

due to security restriction i cant call cloud apis..

vp999 commented 11 months ago

i see few others faced similar issue , and it is getting fixed in https://github.com/jerryjliu/llama_index/pull/7283/commits/5f6f89ed0e89e15013173fe4840218a17ae2cb6c so closing this thread now

yonitjio commented 3 months ago

i see few others faced similar issue , and it is getting fixed in 5f6f89e so closing this thread now

It says This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.?

I'm using 0.10.24 and that fix is not there.