Python 3.7 / PyTorch 1.2

[tomsherborne]

Hi, Do you have any plans to update the code yourself in the future? I've been trying to replicate the results from the paper but I wanted to use a more up to date Python and PyTorch version. Therefore, I've spent some time modifying the model to run with PY3.7 and Pytorch 1.2 (not >1.3.1 as I'm testing using a K40 GPU currently). If you didn't have a plan to make these changes, I have a version with minimal changes from the original code that I can submit as a PR if you'd like to use.

[taolusi]

Hi, Do you have any plans to update the code yourself in the future? I've been trying to replicate the results from the paper but I wanted to use a more up to date Python and PyTorch version. Therefore, I've spent some time modifying the model to run with PY3.7 and Pytorch 1.2 (not >1.3.1 as I'm testing using a K40 GPU currently). If you didn't have a plan to make these changes, I have a version with minimal changes from the original code that I can submit as a PR if you'd like to use.

Hi, Thank you for your work! We would like to use your code if it can achieve the same or even better results than our original code. To this end, we are wondering if you use the official evaluation shown in our CSpider Challenge to test your code since it is easy to operate. If your code does work, we will update the scores in our challenge and show your contribution. If you have any problem with the evaluation process on codalab, feel free to contact me and i will find my co-author Yuefeng @yuefeng-shi to help you.

[tomsherborne]

Thanks, it might take me some time to get through the codalab process but i'll submit a PR when it's done.

[taolusi]

Thanks, it might take me some time to get through the codalab process but i'll submit a PR when it's done.

And you can also submit a PR now if it is ready so that we can review the code first. If you just want to submit a PR, we could test the code also. Please feel free to let us know. If you have any idea about our project, welcome to communicate with us.

[tomsherborne]

Development Set Evaluation using the evaluation script in the repo. I'll proceed with the CodaLab evaluation soon.

                     easy                 medium               hard                 extra                all                 
count                250                  440                  174                  170                  1034                

====================== EXACT MATCHING ACCURACY =====================
exact match          0.316                0.134                0.184                0.018                0.167               

---------------------PARTIAL MATCHING ACCURACY----------------------
select               0.664                0.394                0.618                0.425                0.502               
select(no AGG)       0.676                0.410                0.618                0.425                0.512               
where                0.376                0.247                0.226                0.078                0.245               
where(no OP)         0.385                0.286                0.301                0.117                0.284               
group(no Having)     0.333                0.427                0.512                0.376                0.415               
group                0.333                0.420                0.488                0.365                0.404               
order                0.429                0.297                0.706                0.747                0.556               
and/or               1.000                0.916                0.908                0.895                0.932               
IUEN                 0.000                0.000                0.057                0.125                0.074               
keywords             0.835                0.808                0.703                0.618                0.757               
---------------------- PARTIAL MATCHING RECALL ----------------------
select               0.664                0.393                0.615                0.418                0.500               
select(no AGG)       0.676                0.409                0.615                0.418                0.510               
where                0.380                0.253                0.228                0.061                0.237               
where(no OP)         0.389                0.292                0.304                0.092                0.275               
group(no Having)     0.500                0.427                0.538                0.405                0.442               
group                0.500                0.420                0.513                0.392                0.431               
order                0.545                0.293                0.610                0.728                0.544               
and/or               0.992                0.953                0.969                0.942                0.964               
IUEN                 0.000                0.000                0.048                0.056                0.051               
keywords             0.913                0.804                0.667                0.600                0.755               
---------------------- PARTIAL MATCHING F1 --------------------------
select               0.664                0.394                0.617                0.421                0.501               
select(no AGG)       0.676                0.410                0.617                0.421                0.511               
where                0.378                0.250                0.227                0.069                0.241               
where(no OP)         0.387                0.289                0.303                0.103                0.280               
group(no Having)     0.400                0.427                0.525                0.390                0.428               
group                0.400                0.420                0.500                0.378                0.417               
order                0.480                0.295                0.655                0.737                0.550               
and/or               0.996                0.934                0.937                0.918                0.947               
IUEN                 1.000                1.000                0.052                0.077                0.061               
keywords             0.873                0.806                0.684                0.609                0.756   

[taolusi]

Development Set Evaluation using the evaluation script in the repo. I'll proceed with the CodaLab evaluation soon.

                     easy                 medium               hard                 extra                all                 
count                250                  440                  174                  170                  1034                

====================== EXACT MATCHING ACCURACY =====================
exact match          0.316                0.134                0.184                0.018                0.167               

---------------------PARTIAL MATCHING ACCURACY----------------------
select               0.664                0.394                0.618                0.425                0.502               
select(no AGG)       0.676                0.410                0.618                0.425                0.512               
where                0.376                0.247                0.226                0.078                0.245               
where(no OP)         0.385                0.286                0.301                0.117                0.284               
group(no Having)     0.333                0.427                0.512                0.376                0.415               
group                0.333                0.420                0.488                0.365                0.404               
order                0.429                0.297                0.706                0.747                0.556               
and/or               1.000                0.916                0.908                0.895                0.932               
IUEN                 0.000                0.000                0.057                0.125                0.074               
keywords             0.835                0.808                0.703                0.618                0.757               
---------------------- PARTIAL MATCHING RECALL ----------------------
select               0.664                0.393                0.615                0.418                0.500               
select(no AGG)       0.676                0.409                0.615                0.418                0.510               
where                0.380                0.253                0.228                0.061                0.237               
where(no OP)         0.389                0.292                0.304                0.092                0.275               
group(no Having)     0.500                0.427                0.538                0.405                0.442               
group                0.500                0.420                0.513                0.392                0.431               
order                0.545                0.293                0.610                0.728                0.544               
and/or               0.992                0.953                0.969                0.942                0.964               
IUEN                 0.000                0.000                0.048                0.056                0.051               
keywords             0.913                0.804                0.667                0.600                0.755               
---------------------- PARTIAL MATCHING F1 --------------------------
select               0.664                0.394                0.617                0.421                0.501               
select(no AGG)       0.676                0.410                0.617                0.421                0.511               
where                0.378                0.250                0.227                0.069                0.241               
where(no OP)         0.387                0.289                0.303                0.103                0.280               
group(no Having)     0.400                0.427                0.525                0.390                0.428               
group                0.400                0.420                0.500                0.378                0.417               
order                0.480                0.295                0.655                0.737                0.550               
and/or               0.996                0.934                0.937                0.918                0.947               
IUEN                 1.000                1.000                0.052                0.077                0.061               
keywords             0.873                0.806                0.684                0.609                0.756   

The results on Dev set are close to our original results. You may submit the code as a PR so that we can merge it into ours. Again, please tell us if you meet any problem evaluating using Codalab. This is helpful for us to improve the usability of our evaluation method.

[taolusi]

@tomsherborne Would you like to provide us with your updated code? This could be helpful for follow-up research. Thank you very much!