search

pykeen / benchmarking

📊 Results from the reproducibility and benchmarking studies presented in "Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework" (http://arxiv.org/abs/2006.13365)
MIT License
35 stars 4 forks source link

Why Only Maximum Value? #9

Closed mali-git closed 4 years ago

mali-git commented 4 years ago

Why are we taking only the maximum value?: https://github.com/mali-git/pykeen_experimental_results/blob/e601b40841e334ad5d0c98e0d55c12d93a4fc36c/ablation/collate.py#L207

This way, we loose information about the other configurations. For each configuration, we already export only the best results, and we should show all of them

mali-git commented 4 years ago

In the current version of results.tsv, we don't have created a column indicating whether inverse triples have been added or not. For this reason, it seems that we have the same configurations twice, but actually these are different configurations.

mali-git commented 4 years ago

When have run all experiments, for each model + dataset, we will have 28 configurations. So, for example in the dataset plots, for each model we would need to display 28 points within its boxplot.

mali-git commented 4 years ago

@cthoyt What is the status here?

cthoyt commented 4 years ago

This is fixed