Main KPI or list of KPIs in challenge tasks

[denkv]

We need to be able to specify main KPI for challenge tasks, to implement leaderboards.

Proposed name: hobbit:hasMainKPI.

Another possibility is to specify a list of KPIs instead of that for tie breaks and just use it instead - we probably would need such a list anyway.

[jsaveta]

Maybe a list of KPIs is better.

[MichaelRoeder]

RDF offers a sequence https://www.w3.org/TR/2014/REC-rdf-schema-20140225/#ch_seq

An example could look like:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix bench: <http://example.org/bench#> .
@prefix chal: <http://w3id.org/hobbit/challenges#> .

# --- Challenge ---

chal:challenge  a   hobbit:Challenge;
    rdfs:label "The first example challenge!!!"@en;
    rdfs:comment    "This is an example for a challenge."@en;
    hobbit:executionDate "2016-12-24"^^xsd:date;
    hobbit:publicationDate "2016-12-26"^^xsd:date;
    hobbit:organizer "Michael Roeder"@en .

# --- Challenge Task 1 ---

chal:challenge_Task1 a hobbit:ChallengeTask;
    rdfs:label "Example task 1"@en;
    rdfs:comment    "Task 1 of my example challenge."@en;
    hobbit:isTaskOf chal:challenge;
    hobbit:involvesBenchmark bench:lance
    hobbit:orderedKPIs  chal:challenge_Task1_KPIs .

chal:challenge_Task1_KPIs a rdf:Seq
    rdf:_1  bench:fmeasure ;
    rdf:_2  bench:recall ;
    rdf:_3  bench:precision .

# --- Benchmark ---

bench:lance a   hobbit:Benchmark;
    rdfs:label "Lance Benchmark"@en;
    rdfs:comment    "LANCE is a benchmark for the assessment of Instance Matching techniques and systems for Linked Data data that are accompanied by a schema."@en;
    ...
    hobbit:measuresKPI bench:precision ;
    hobbit:measuresKPI bench:recall ;
    hobbit:measuresKPI bench:fmeasure .

bench:precision a hobbit:KPI;
    rdfs:label "Precision"@en;
    rdfs:comment    "Precision = TP / (TP + FP)"@en;
    rdfs:range  xsd:float .

bench:recall a hobbit:KPI;
    rdfs:label "Recall"@en;
    rdfs:comment    "Recall = TP / (TP + FN)"@en;
    rdfs:range  xsd:float .

bench:fmeasure a hobbit:KPI;
    rdfs:label "F-measure"@en;
    rdfs:comment    "F-measure is the harmonic mean of precision and recall."@en;
    rdfs:range  xsd:float .

I simply called the property hobbit:orderedKPIs in the example, but I woule be happy if we can find a better name. To me, it looks like a good solution, since we can express the order without introducing the usage of blank nodes.

[denkv]

We need to decide of some name for that, hobbit:significantKPIs, :scoringKPIs, :rankingKPIs maybe? :orderedKPIs is almost good, but relays no intent.

[MichaelRoeder]

significant might have a different meaning in evaluation ;) :rankingKPIs looks good to me.