KnowledgeNet is a benchmark dataset for the task of automatically populating a knowledge base (Wikidata) with facts expressed in natural language text on the web. KnowledgeNet provides text exhaustively annotated with facts, thus enabling the holistic end-to-end evaluation of knowledge base population systems as a whole, unlike previous benchmarks that are more suitable for the evaluation of individual subcomponents (e.g., entity linking, relation extraction).
For instance, the dataset contains text expressing the fact (Gennaro Basile; RESIDENCE; Moravia), in the passage: "Gennaro Basile was an Italian painter, born in Naples but active in the German-speaking countries. He settled at Brünn, in Moravia, and lived about 1756..."
For a description of the dataset and baseline systems, please refer to our EMNLP paper.
To visualize the dataset you can dowload this file and open it with a browser. It contains a human readable tabular version of the dataset.
| System | Link F1 | Text F1 |
|---|---|---|
| Human | 0.822 | 0.878 |
| Diffbot Joint Model | 0.726 | 0.810 |
| DYGIE++ | N/A | 0.754 |
| Diffbot NER, coref, linker + Matching the Blanks (relations) | 0.695 | 0.737 |
| Diffbot Open Relation Extraction (no training data for relations) | 0.546 | 0.560 |
| KnowledgeNet Baseline 5 (Baseline 4 + BERT for relations) | 0.504 | 0.688 |
| KnowledgeNet Baseline 4 (Baseline 3 + noisy candidate facts) | 0.491 | 0.621 |
| KnowledgeNet Baseline 3 (Baseline 2 + external information) | 0.362 | 0.545 |
| KnowledgeNet Baseline 2 (Baseline 1 + Huggingface coref) | 0.342 | 0.554 |
| KnowledgeNet Baseline 1 (Spacy NER, Diffbot linker, Bi-LSTM for relations) | 0.281 | 0.518 |
The training set is available at train.json. Each document contains a number of passages. A passage represents a sentence considered for annotation. Here is an example document:
{
"fold": 2,
"documentId": "8313",
"source": "DBpedia Abstract",
"documentText": "Gennaro Basile\n\nGennaro Basile was an Italian painter, born in Naples but active in the German-speaking countries. He settled at Brünn, in Moravia, and lived about 1756. His best picture is the altar-piece in the chapel of the chateau at Seeberg, in Salzburg. Most of his works remained in Moravia.",
"passages": [
{
"passageId": "8313:16:114",
"passageStart": 16,
"passageEnd": 114,
"passageText": "Gennaro Basile was an Italian painter, born in Naples but active in the German-speaking countries.",
"exhaustivelyAnnotatedProperties": [
{
"propertyId": "12",
"propertyName": "PLACE_OF_BIRTH",
"propertyDescription": "Describes the relationship between a person and the location where she/he was born."
}
],
"facts": [
{
"factId": "8313:16:30:63:69:12",
"propertyId": "12",
"humanReadable": "<Gennaro Basile> <PLACE_OF_BIRTH> <Naples>",
"annotatedPassage": "<Gennaro Basile> was an Italian painter, born in <Naples> but active in the German-speaking countries.",
"subjectStart": 16,
"subjectEnd": 30,
"subjectText": "Gennaro Basile",
"subjectUri": "http://www.wikidata.org/entity/Q19517888",
"objectStart": 63,
"objectEnd": 69,
"objectText": "Naples",
"objectUri": "http://www.wikidata.org/entity/Q2634"
}
]
},
{
"passageId": "8313:115:169",
"passageStart": 115,
"passageEnd": 169,
"passageText": "He settled at Brünn, in Moravia, and lived about 1756.",
"exhaustivelyAnnotatedProperties": [
{
"propertyId": "11",
"propertyName": "PLACE_OF_RESIDENCE",
"propertyDescription": "Describes the relationship between a person and the location where she/he lives/lived."
},
{
"propertyId": "12",
"propertyName": "PLACE_OF_BIRTH",
"propertyDescription": "Describes the relationship between a person and the location where she/he was born."
}
],
"facts": [
{
"factId": "8313:115:117:129:134:11",
"propertyId": "11",
"humanReadable": "<He> <PLACE_OF_RESIDENCE> <Brünn>",
"annotatedPassage": "<He> settled at <Brünn>, in Moravia, and lived about 1756.",
"subjectStart": 115,
"subjectEnd": 117,
"subjectText": "He",
"subjectUri": "http://www.wikidata.org/entity/Q19517888",
"objectStart": 129,
"objectEnd": 134,
"objectText": "Brünn",
"objectUri": "http://www.wikidata.org/entity/Q14960"
},
{
"factId": "8313:115:117:139:146:11",
"propertyId": "11",
"humanReadable": "<He> <PLACE_OF_RESIDENCE> <Moravia>",
"annotatedPassage": "<He> settled at Brünn, in <Moravia>, and lived about 1756.",
"subjectStart": 115,
"subjectEnd": 117,
"subjectText": "He",
"subjectUri": "http://www.wikidata.org/entity/Q19517888",
"objectStart": 139,
"objectEnd": 146,
"objectText": "Moravia",
"objectUri": "http://www.wikidata.org/entity/Q43266"
}
]
}
]
}The official evaluation script is also available for download and can be used to evaluate a system using the training set (via cross-validation). The script takes a gold standard file (e.g., train.json) and a prediction file (which needs to be produced by the system). The prediction file should look exactly like the gold standard file (same documents and fields), except for the contents of facts (which should contain the facts predicted by the system).
usage: evaluator.py [-h] [-e {span_exact,span_overlap,uri}] [-c]
[-a ANALYSISPATH] [-f {1,2,3,4,5,-1}]
goldFile predictionFile
positional arguments:
goldFile path of the KnowledgeNet file with the gold data
predictionFile path of the KnowledgeNet file with the predicted data
optional arguments:
-h, --help show this help message and exit
-e {span_exact,span_overlap,uri} choose the evaluation method: span-exact vs span-overlap vs uri
-c print raw counts of tp/fn/fp for prec/rec/F1 metrics
-a ANALYSISPATH folder to store error analysis and results files
(default=no analysis).
-f {1,2,3,4,5,-1} folds to evaluate. Default is 4. Choose -1 to evaluate on all the folds.The prediction file has to keep the same unique identifiers and attributes for the given documents and passages. Each new fact must contain the following attributes that are needed to run the evaluation script:
subjectStartsubjectEndobjectStartobjectEndsubjectUri (can be empty)objectUri (can be empty)propertyIdA factId will be automatically generated from these attributes.
Two facts are considered the same when they have the same property, and there is a match between the values for subject and object.
We consider three different methods to establish if there is a match:
span_overlap): there is an overlap between the character offsets (default in the evaluation script)span_exact): the character offsets are exactly the sameuri): Wikidata URIs are the same (only applies to facts that have URIs for both the subject and the object)In order to facilitate error analysis the script creates a simple html file for browser visualization. It can be enabled using the option -a <path to the analysis folder>.
To preserve the integrity of the results, we have released the test set (fifth fold) without annotations (test-no-facts.json). To evaluate the results of your system and (optionally) add your system to the leaderboard, please send an email with your prediction file to filipe[at]diffbot[dot]com.