inab / benchmarking-data-model

OpenEBench Benchmarking Data Model repository
Creative Commons Attribution Share Alike 4.0 International
2 stars 7 forks source link

Suggestion of new attribute in Challenge schema #61

Closed javi-gv94 closed 5 years ago

javi-gv94 commented 6 years ago

In order to help results visualization/classification, we could include a new attribute in the challenge schemas that indicates how to optimize the metrics in that challenge (e.g. maximize both metrics - topRight corner of the plot, maximize one/minimize the other - bottomRight corner...) That is required in the visualization chart to apply the square/diagonal quartiles methods.

We could set a property like: _"optimal_performace": enum['top_right', 'bottom_right', 'bottomleft'...]

Or, for more completeness, set a 'ranking' for each corner. e.g.: _"chart_optimization": { 'top_right':1 'bottom_right':2 'top_left':3 'bottomleft':4 }

jmfernandez commented 6 years ago

I think we should keep representation layer separated from the data model. But, it is also true that some hints for the visual representation can be helpful.

So, from my point of view, the challenge could store those hints as a related dataset

jlgelpi commented 5 years ago

Decission: To define a specific document (Json) to include the necessary parameters for any process implying metrics assessment, analysis, etc. That would include docs for adaptation to the workbench, etc. Define a new type of Dataset (Process Dataset?) to include this.

JuergenHaasSIB commented 5 years ago

why not add a field to each metrics? the community announces what best-performing translates to.

jmfernandez commented 5 years ago

Added representation_hints to Metrics in commit 002eff99a565d97318a1fcdb5e25fcbefef97c8e