jh-RLI
commented
1 month ago We have already implemented a qualitative comparison that finds similarities by comparing certain descriptors/attributes that were assigned to a scenario when the scenario bundle was created by the user. These values are linked to the oeo classes. The system then compares the "base" scenario with the others and gives a green colour to all attributes that are also included in the base scenario and a red colour to the attributes that are not included.
Quantitative comparison is much more challenging: The first step we took to formulate an approach was to implement user-made data-annotations via the oemetadata. The annotation is done in 3 different metadata fields so far: For dataset as a unit, we are annotating "subjects". Subject annotations describe the topic the dataset is about, like "GHG emissions". As we work with relational data formats, that is why we also implemented annotations for the tables schema descriptions (that are part of the metadata). These annotations can be applied to each column defined in a table. There are two types of annotations available: 1. "is_about": These annotations enable the user to specify the column name. For example, a user can annotate the year column with "scenario years" to specify that this column includes all scenario years that have been used. 2. the "value_referance" is used to enable specific value annotations that are available in the column. For example the column "country_code" could be annotated with the Country full name and so on.
Now we need to find enabling technologies that can query our SQL database using annotations. The "ontop" software is build to translate sparql to sql queries by using mappings. These mappings have to be available for each table that is queried this way in advance. This makes it challenging to enable our approach for all available tables dynamically, the manual step to create the mapping is always needed. But this approach enables us to do what we want and is good enough for demonstration of the approach.
Since we know how to retrieve the data using our annotations instead of plain sql queries, we can progress with the next step. We want to compare all tables (e.g.) all output tables included in a specific scenario with all output tables from another scenario. Now the challenge is to properly transform the data into a format that can be visualized.
We assume that we can harmonize the data structure by querying the data tables based on the is_about annotation. Users who annotate table columns equally should have comparable data in these columns. Here we have to do some testing if our assumption will work out.
From here on we can either use all unique values from specific annotated columns, like sector division or greenhouse gas emissions, to filter our data and interactively change the visualization and compare other values across the scenarios. Another approach would be to attempt to calculate specific "Kennzahlen" that can be compared across scenarios. I assume we can attempt to calculate these "Kennzahlen" if specific annotations are available.
This for now, i will document how we decided and what we implemented.
What is necessary for scenario comparisons? Collect requirements here before creating several issues for the implementation.