search

ryanzhumich / editsql

MIT License
202 stars 68 forks source link

Some questions about repeating the results on SParC. #16

Closed RandomTuringDuck closed 4 years ago

RandomTuringDuck commented 4 years ago

I want to reproduce the result of SParC on my server, and i only download the code and run the run.sh following the README. The model 'saved_42' is chosen to get the final result. 2020-03-11 22-10-57屏幕截图 But the final result is only 46.0, which is lower than the results in the paper. Although I repeat a lot of times, I get some similar results like 45.8 or 45.5. 2020-03-11 22-19-52屏幕截图 Maybe are the experimental environments different? or you use some the different parameters, like learning rate or something. I hope you can give me some suggestions.

runzewang commented 4 years ago

I run this code 5 times, and the average question accuracy was 45.4% and the average interaction accuracy was 26.46%. If you reproduce the results, please give me some suggestions. Thank you!

huybery commented 4 years ago

@RandomTuringDuck Have you reproduced the results on Sparc ?

Param-Raval commented 4 years ago

When I train the model on SParC and run the testing on the saved model using test_sparc_editsql.sh for some reason it throws this error Unexpected key(s) in state_dict: "utterance_attention_module.query_weights". Can you tell me what's the issue @ryanzhumich ?