In recent years, a significant number of question answering (QA) systems that retrieve answers to natural language questions from knowledge graphs (KG) have been introduced. However, finding a benchmark to appropriately measure the quality of a question answering system is a difficult task because of (1) the high degree of variation with respect to the fine-grained properties among the available benchmarks; (2) the available benchmarks are static by nature while the KG evolving over time; (3) the existing benchmarks in the literature target only a handful of KGs. We introduce Maestro, an on-the-fly benchmarking system for QA over KGs systems that can be used to build a QA benchmark over any KG. The benchmark generated by Maestro is guaranteed to cover all the properties of the natural language questions and queries that were encountered in the literature as long as the targeted KG’s topology demonstrates these properties. Moreover, Maestro generates benchmarks that follow the same standards and complexity distribution to solve the variation of the fine-grained properties among generated benchmarks.
Maestro requires the following development kits and liberaries. You can download the liberaries with the system.
pom
file, use Maven for this purpose.]main.py
file and the Prerequisites for this python file can be found in the requirements.txt
file.settings
package.
projectFolder
│
└─── Maestro.jar
│
|─── data
│ |─── benchmark.json
│ |─── src
│ │───
│ └───
│
└─── lib
└─── ... All .jar files
* __Run CBench.jar:__ Using the command ``` java -jar "PATH/TO/projectFolder/Maestro.jar" ```, run the project.
## Support
Please raise potential bugs on github. If you have a research related question, please send it to this email(abdelghny.orogat@carleton.ca)