search

naver / sqlova

Apache License 2.0
632 stars 168 forks source link

Train lf accuracy higher than execution accuracy #9

Closed vincentlux closed 5 years ago

vincentlux commented 5 years ago

Hi,

When running the script, it is surprised to see that after several iterations (<3), the training logical form accuracy becomes higher than the training execution accuracy, which confuses me a lot.

May I ask what is the reason behind it? I am confused for several days and could not figure it out. Thank you for any help in advance!

whwang299 commented 5 years ago

Hi @vincentlux

This might happen due to instability of DBEngine class which activates except in following code. https://github.com/naver/sqlova/blob/7a8636d6d0c9a46b32b6d00eccaafeea1aab5e3b/sqlova/utils/utils_wikisql.py#L1646-L1670

DBEngine class is modified from SQLNet code which is based on original WikiSQL evaluation code (I guess).

I'll check whether this still happens when DBEngine from WikiSQL evaluation code is employed.

Thanks!

Wonseok

vincentlux commented 5 years ago

Hi Wonseok,

Thanks for the quick response! This is interesting and I will also check with using the option you suggested. Thanks!

whwang299 commented 5 years ago

@vincentlux I found the problem. It is caused by the wrong ordering of predicted where-column.

For example, when the ground truth is {'sel': 1, 'conds': [[6, 0, 'at&t center 18,797'], [0, 2, 57]], 'agg': 0} the previous prediction in train function could be {'agg': 0, 'sel': 1, 'conds': [[0, 0, 'at&t center 18,797'], [6, 2, '57']]}

Hence, the execution accuracy of the model over train-set returned from the train function has been under-estimated. During the test, there is no problem as this issue is caused by the use of ground-truth during the training which makes the debugging difficult.

I have updated the code. Please let me know if the problem still happens.

Wonseok