Using Open source LLM models in SQL Chain

[kb-shiva]

Using Open source LLM models in SQL Chain

Is it possible to use open source LLM models in SQL chain ? I have tried using tapex/Flan models in SQL Chain, but getting a serialization error on dict[] classes.

Error:

 File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for SQLDatabaseChain
__root__ -> llm
  **_value is not a valid dict (type=type_error.dict)_**

Are there any samples/snippets are available for using open source LLM models in SQL Chain ?

Sample code snippet I tried that is throwing the error:

tokenizer = TapexTokenizer.from_pretrained("microsoft/tapex-base-finetuned-wtq")
model = BartForConditionalGeneration.from_pretrained("microsoft/tapex-base-finetuned-wtq")
chain = SQLDatabaseChain(llm=model, database=db, verbose=True)
chain.run("context query ?")

Suggestion:

No response

[BrettlyCD]

@kb-shiva you could try accessing the model through the HuggingFaceHub. Here is the documentation. I haven't tried that specific model, but have had success with others. If you have a HuggingFace API Token you should be able to test this method out:

#imports
from langchain import HuggingFaceHub
from langchain import SQLDatabase, SQLDatabaseChain

#get api token from .env file
load_dotenv()
hf_api_token = os.getenv('hf_token')

#add path to HF repo
repo_id = 'microsoft/tapex-base-finetuned-wtq'

#establish llm model
llm = HuggingFaceHub(repo_id=repo_id, huggingfacehub_api_token=hf_api_token, model_kwargs={"temperature": 0.5, "max_length": 64})

And then add that to your chain. Hope that helps!

[kb-shiva]

Hey @BrettlyCD , Thanks for the reply. I tried building a plain llmchain() and it worked fine. Any chance you have some samples for building a sqldatabasechain() ?

I tried some open source models, but itcould generate a sql properly and it failed with sqlite incorrect sql error. It would be great if you can share any open source model and a sample code for SQL Database chain.

[BrettlyCD]

@kb-shiva

It sounds like we've been struggling with the same challenges if I'm understanding correctly. I've been able to get the process to work, meaning the prompt is accurately going to the model, but I'm still not getting the results I'm looking for.

It's been useful for me to also use Chainlit which has the ability to drill into the prompt template and see exactly what the model received. Here is my code for an "out-of-the-box" SQLDatabaseChain. I'm not actually sure if you need to add the prompt template, but I did some minor tweaks to the wording in a few spots:

import os
from dotenv import load_dotenv
import chainlit as cl
from langchain import PromptTemplate, SQLDatabase, SQLDatabaseChain, HuggingFaceHub

load_dotenv()
hf_api_token = os.getenv('hf_token')

repo = 'tiiuae/falcon-7b-instruct'

template = """You are a SQLite expert. Given an input question, first create a syntactically correct SQLite query to run, then look at the results of the query and return the answer to the input question.
            Unless the user specifies in the question a specific number of examples to obtain, query for at most {top_k} results using the LIMIT clause as per SQLite. You can order the results to return the most informative data in the database.
            Never query for all columns from a table. You must query only the columns that are needed to answer the question. Wrap each column name in double quotes (") to denote them as delimited identifiers.
            Pay attention to use only the column names you can see in the tables below. Be careful to not query for columns that do not exist. Also, pay attention to which column is in which table.
            Pay attention to use date('now') function to get the current date, if the question involves "today".

            Use the following format:

            Question: Question here
            SQLQuery: SQL Query to run
            SQLResult: Result of the SQLQuery
            Answer: Final answer here

            Only use the following tables:
            {table_info}

            Question: {input}"""

@cl.langchain_factory(use_async=False)
def factory():
    prompt = PromptTemplate(template=template, input_variables=["input","table_info","top_k"])

    db = SQLDatabase.from_uri("sqlite:///" + '<db_path>')

    llm = HuggingFaceHub(repo_id=repo, huggingfacehub_api_token=hf_api_token, model_kwargs={"temperature": 0.5, "max_length": 200})

    sql_chain = SQLDatabaseChain.from_llm(prompt=prompt, llm=llm, db=db, verbose=True)

    return sql_chain

This works for really specific and simple questions:

But breaks down when it gets more complicated. Part of this could be the result of using an open source model but you can copy the exact prompt into Chat-GPT and often get the same result. From what I understand, the prompt is providing all the schema information in that database and asking the LLM to find the best query to answer the question. This can be a tough task when there is a lot of tables/colums in the schema.

You could also try using the SQLDatabaseSequentialChain which runs the model by first asking which tables are most relevant to the question, returning those tables, and then running the normal SQLDatabaseChain on those specific tables. Or the SQLDatabaseAgent which walks the model more through a chain-of-thought process to handle more general questions and recover from errors. I haven't had success with either, but I'm also just getting into this so it could be user error.

The other thing I'm going to try exploring over the next few weeks is a more custom approach like this blog walks through. Seems promising to tweak things here and there to respond best to your data.

I'll keep you updated on what I learn. I'd love to learn from your progress as well. :)

[kb-shiva]

Thanks @BrettlyCD - Will let you know incase if I get a workable solution.

[pavan-growexxer]

Hello @kb-shiva & @BrettlyCD, I am also struggling with the same tasks to setup an opensource llm with langchain to query snowflake. I have successfully setup the model & snowflake db, but it is giving me output parser error with defog/sqlcoder model. Can you guys help me out with how i can run it successfully. I am using SQLDatabaseToolkit. So which langchain module should i use for my usecase (query snowflake with natural language) SQLDatabaseSequentialChain, SQLDatabaseAgent or other?

[sathyapriyaa13]

Guys @kb-shiva @pavan-growexxer @BrettlyCD I'm also stuck with the same problem. I want to leverage SQLDatabaseChain using Opensource llm's. Any other solutions might help.

[virux99]

same issue, with open AI model, the big databases doesn't seems cost effective and feasible. so i wonder if we can use head of open source model like llma2, :/

[dosubot[bot]]

Hi, @kb-shiva,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, you encountered a serialization error with dict[] classes when trying to use open source LLM models in SQL Chain. BrettlyCD suggested accessing the model through the HuggingFaceHub and provided a code snippet for using it in SQL Database Chain. Other users also expressed similar struggles and sought help in leveraging SQLDatabaseChain using open source LLMs. The discussion also touched on using SQLDatabaseSequentialChain and SQLDatabaseAgent for different use cases. The issue has been resolved with BrettlyCD's suggestion to access the model through the HuggingFaceHub and provided a code snippet for using it in SQL Database Chain.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself or the issue will be automatically closed in 7 days.

Thank you!