Low accuracy in predicting SQL using RESDSQL on my dataset

[VirendraSttl]

Hello everyone,

I hope you're doing well. I encountered an issue while using RESDSQL for predicting SQL on my dataset. Despite following all the recommended steps, I'm observing an accuracy range of only 30-40%. I would greatly appreciate any suggestions or insights on increasing the predictions' accuracy.

Thank you in advance for your assistance!

[lihaoyang-ruc]

Hi! Have you fine-tuned RESDSQL on your dataset? Or did you only use the checkpoints we provided to perform inference on your dataset?

[VirendraSttl]

I have been utilizing the provided checkpoints in RESDSQL to enhance my work. However, I am uncertain about the process of fine-tuning RESDSQL on my own dataset. Could you kindly provide guidance on how to proceed with fine-tuning RESDSQL using my specific dataset?

[lihaoyang-ruc]

RESDSQL has been fine-tuned on Spider. Therefore, you should prepare your dataset in the same format as it (its home page https://yale-lily.github.io/spider).

In fact, most Text-to-SQL datasets organize their data in Spider's format (e.g., Dr. Spider, CSpider, BIRD, Kaggle-DBQA, etc.).

[VirendraSttl]

I had already set my dataset in the format of Spider (i.e. tables.json)

[lihaoyang-ruc]

Just tables.json is not enough.

To train RESDSQL on your dataset, you have to prepare at least three files (Take Spider's file as an example):

• database, a folder where the sqlite databases are saved.
• train_spider.json, a json file that contains pairs of training data, each of them should contain three fields: db_id, query, and question.
• tables.json, a json file that describes the schema of all databases.

To run inference and evaluation, you should prepare a separate dev_gold.sql file containing the gold SQL query and its corresponding db_id.

[VirendraSttl]

Okay, I'll try this solution.

BTW I have a question do I need to train the model every time whenever I change my dataset? Is there any way RESDSQL will generate an SQL query on the hidden test set?

[lihaoyang-ruc]

No, if your training set and test set have the same (or similar) distribution, it can be naturally generalized to the hidden test set without additional training.

[VirendraSttl]

Okay.

I followed your training steps and noticed that both train_spider.json and dev.json were required. However, I am a bit confused about their differences. Are they essentially the same file with different names, or do they serve distinct purposes in the training process?

[lihaoyang-ruc]

They are different files. train_spider.json is the training set, and dev.json is the development set.

[lihaoyang-ruc]

We use dev.json to select the best checkpoint during fine-tuning.

[VirendraSttl]

So can I use the same dev.json file? or Do I need to create it separately as per my dataset?

[lihaoyang-ruc]

My suggestion would be to create a separate dev.json so that you can evaluate the performance of the model on unseen data.

[lihaoyang-ruc]

If you're training and evaluating your model on the training set, I don't think it makes sense because the model will memorize your training data to quickly reach (close to) 100% accuracy.

[VirendraSttl]

You mean, train_spider.json and dev.json are the same as we are splitting our data into two sets i.e. train set and test set

[lihaoyang-ruc]

Yes