Enable automatic benchmarking

AskNowQA / LC-QuAD

A data set of natural language queries with corresponding SPARQL queries

GNU General Public License v3.0

90 stars 30 forks source link

Enable automatic benchmarking #3

Open geraltofrivia opened 6 years ago

geraltofrivia commented 6 years ago

Is your feature request related to a problem? Please describe. Need a script to seamlessly use a QA solution, and benchmark its results over LC-QuAD

Describe the solution you'd like

Require them to open an API which returns answers or SPARQLS given a question.
Use over-arching platforms like Project Hobbit, but this requires dockerized systems and other added constraints.
What else?

geraltofrivia commented 6 years ago

Requesting suggestions from @mohnish-rygbee @debayan @saist1993

RicardoUsbeck commented 5 years ago

Hi, have you advanced in this point or do are you still searching for suggestions?

RicardoUsbeck commented 5 years ago

I will just leave a link here in case I forget this topic. https://github.com/dice-group/TextComparisonEvaluationTool/blob/43c263d905f8b05071c6034d484f9fe199ba2e8a/src/main/java/Engines/Run/Pipeline.java#L210 Source code from this link can be reused to call GERBIL/GERBILQA automatically from code. That is, all evaluation of a system (provided it has a webservice) is done via the GERBIL platform, the results are archived there but you can get all the results as JSON-LD as well as a citable URI.