Anamolies in generated queries

[udaramrdc]

@tscholak

1. For input text : orders placed since 2016-01-5 by Debra Output : select t1.order_id from orders as t1 join customers as t2 on t1.customer_id = t2.customer_id where t2.first_name = "Debra" and t2.last_name = "Dennis"

Model is adding extra lastname Dennis that is neither provided in text nor it is there in database

2. Text : Cumulative item price for order received in year 2016 Select avg(list_price) from order_items where order_id in (select t1.order_id from orders as t1 join order_items as t2 on t1.order_id = t2.order_id where ''year'= 2016)

Model is adding extra column 'year' which is not there in database....

I am trying to generate queries for a bikestore database and I am using 10 gb trained model of picard. Trying all these examples without training/fine-tuning the model on this database.

What could be the possibile reason for above anomalies? How can these be removed?

[tscholak]

@udaramrdc

1. These things happen. Sometimes neural models hallucinate things. Picard cannot prevent this specific mistake because there is nothing syntactically or semantically wrong with the SQL. We also don't prevent the model from producing values that are not in the database. If we did, then a question about an unknown entity will still produce a result.
2. This issue should be caught by picard if you turn it on. If there isn't a column year, it will not be predicted. Without picard, this is just another expected failure mode of the neural model and cannot be prevented (just made more and more unlikely with additional fine-tuning).

[udaramrdc]

Hi @tscholak Thanks for answering my earlier queries. I need additional clarification We are trying the picard model and wanted to study the top 50 outputs/queries generated for a given text. As per my understanding,I have to make the following changes in the configurations -

1. Set num_return_sequences = 50
2. Set num_beams = 51 (as per my understanding this value has to be great than num_return_sequences)

Kindly confirm whether my understanding is correct. Please suggest in case I am missing something

[tscholak]

Hi,

To answer you question: num_return_sequences <= num_beams.

Btw, picard is not a model. it's a method for constrained inference on top of an existing model. it is optional. you can use the underlying model (usually T5) without it. if you turn picard on, inference will be constrained to sequences that picard deems correct.