search

allenai / allennlp

An open-source NLP research library, built on PyTorch.
http://www.allennlp.org
Apache License 2.0
11.77k stars 2.25k forks source link

Ask for Semantic Parsing Baseline of DROP #3720

Closed tonyliangli closed 4 years ago

tonyliangli commented 4 years ago

Hi,

I'm interested in DROP dataset, especially the semantic parsing baseling part and would like to do some further research based on it. Is it possible to add this baseline in allennlp or share the original implemetation code?

Thanks a lot.

matt-gardner commented 4 years ago

For the model, we just used our wikitables semantic parser, which is in our semantic parsing code. This code now lives in the allennlp-semparse repository, though there may be some bugs right now as we're transitioning to allennlp v1.0. You could instead just use version 0.9, where the code is still in allennlp.

The challenging thing (and the part that didn't work very well) was getting the paragraphs into a table format that we could use with the parser, and I think @pdasigi also modified the language that was used, to try to better handle the SRL / OpenIE output. @pdasigi would be able to tell you more about what code is available. I know there was a drop-baselines repository somewhere, but it was probably private. Pradeep, do you remember?

In any event, if you want to do research on semantic parsing for DROP, I would strongly recommend using instead the neural symbolic reader: https://openreview.net/forum?id=ryxjnREFwH. Not sure if they have code available yet, but I would ask them for it. They built a semantic parser for DROP that actually works.

matt-gardner commented 4 years ago

Or the other option is to use neural module networks: https://openreview.net/forum?id=SygWvAVFPr. That code will be available very soon - @nitishgupta is preparing a repository to share. As far as performance goes on DROP, the neural symbolic reader is better. Both systems have tradeoffs, and both directions could use more research (and both are substantially better than our initial semantic parsing baselines).

nitishgupta commented 4 years ago

The code neural module networks for drop is now public and resides at: https://nitishgupta.github.io/nmn-drop/. Let me know if you have any questions.

pdasigi commented 4 years ago

@tonyliangli The code for the semantic parsing baseline for DROP is here: https://github.com/pdasigi/drop-semparse-baseline

tonyliangli commented 4 years ago

Thanks a lot for the detailed reply and quick response. Really helpful.