Closed oakor closed 1 year ago
tom68-ll
commented
1 year ago Hello author @Bahuia , we are also troubled by this issue and hope to receive your response as soon as possible. We have also noticed that the results of SFNET reported in Table 1 of the paper are even better than those of the full data-trained ORACLE on WikiSQL. We would appreciate it if you could provide further insights into this matter.
Bahuia
commented
1 year ago Hi author! First of all, I'd like to express my gratitude for your hard work on this project. Your dedication is greatly appreciated. Recently, I encountered an issue while testing your code on WikiSQL. The results were not as satisfying as I had hoped. I attempted to diagnose and rectify the problem myself but regrettably, I was unable to find a solution. To my surprise, I discovered that another user has also faced the same issue and it remains unresolved. I apologize for any inconvenience, but I was wondering if you could spare some time to look into this matter. Your expertise and insights would be invaluable in resolving this challenge. Thank you once again for your contribution and assistance. I look forward to your response and any potential solutions you may have.
Over the past two days, I have started working on resolving this matter, and I anticipate providing a response within the next two to three weeks.
Bahuia
commented
1 year ago Hi author! First of all, I'd like to express my gratitude for your hard work on this project. Your dedication is greatly appreciated. Recently, I encountered an issue while testing your code on WikiSQL. The results were not as satisfying as I had hoped. I attempted to diagnose and rectify the problem myself but regrettably, I was unable to find a solution. To my surprise, I discovered that another user has also faced the same issue and it remains unresolved. I apologize for any inconvenience, but I was wondering if you could spare some time to look into this matter. Your expertise and insights would be invaluable in resolving this challenge. Thank you once again for your contribution and assistance. I look forward to your response and any potential solutions you may have.
Hi, I have addressed the issue of the wikisql results. The bug primarily stems from the to_examples function located in sfnet/utils.py. The input enhancement with the linked schema performed by IRNet for the Spider dataset is not applicable to the WikiSQL dataset. https://github.com/Bahuia/SSCL-Text2SQL/blob/a6a11b918d169c8bfbf2e6df387d76dde85688fe/sfnet/utils.py#L343-L393 In my initial release, I mistakenly implemented it for both WikiSQL and Spider, resulting in unexpected outcomes. Now you can directly run the train_wikisql.sh script to achieve the results reported in the paper.
tom68-ll
commented
1 year ago Hi author! @Bahuia
Thank you very much for your fix. However, we have some questions regarding the results on WikiSQL and would appreciate your insights.
In the revised WikiSQL results, we noticed that all values for the BWT metric are positive, which we also observed in our own replication. This seems somewhat unexpected and inconsistent with the results reported in the paper, unless the model demonstrates a very strong generalization ability. https://github.com/Bahuia/SSCL-Text2SQL/blob/e55e111e2e916b0a46f08c12cbd0d710049ecc91/wikisql.log#L12543
Simultaneously, we noticed that on WikiSQL, SFNET's performance even surpasses the ORACLE method, which appears to be a quite intriguing discovery. However, the reasons behind this phenomenon are not entirely clear to us. Could you please provide further insights on this matter? https://github.com/Bahuia/SSCL-Text2SQL/blob/e55e111e2e916b0a46f08c12cbd0d710049ecc91/wikisql.log#L12544
Hi author! First of all, I'd like to express my gratitude for your hard work on this project. Your dedication is greatly appreciated. Recently, I encountered an issue while testing your code on WikiSQL. The results were not as satisfying as I had hoped. I attempted to diagnose and rectify the problem myself but regrettably, I was unable to find a solution. To my surprise, I discovered that another user has also faced the same issue and it remains unresolved. I apologize for any inconvenience, but I was wondering if you could spare some time to look into this matter. Your expertise and insights would be invaluable in resolving this challenge. Thank you once again for your contribution and assistance. I look forward to your response and any potential solutions you may have.