Simplify the input database structure

[datejada]

Discussed in https://github.com/spine-tools/SpineOpt.jl/discussions/581

^{Originally posted by **nhniina** March 24, 2023} The number of classes and parameters in the input database structure feels a bit overwhelming for many new users. Particularly, there seem to be very many classes related to the temporal and stochastic structure. Are there ways to simplify or clarify the structure without reducing the functionalities or making it more difficult to use the advanced features of SpineOpt? One option could be to categorise the classes in the DB Editor. This would probably require additional features to the DB Editor itself. Some classes are there to introduce certain variables whereas some classes are "just" to store parameters. However, it is not easy to grasp that idea currently. In addition, there are multiple classes related to advanced temporal and stochastic structures as well as user constraints. Thus, there could be categories, for example, for - parameters - variables - advanced temporal and stochastic structures - user constraints I suppose there are cases where a class belongs to multiple categories. How to handle those cases? **Ideas for the classes related to the temporal and stochastic structure** 1. Create a new class `group` to group connections, nodes and units with similar temporal and stochastic structures. Create also new classes `connection__group`, `node__group` and `unit__group`. Replace corresponding `connection/node/unit__...` classes with - `group__investment_stochastic_structure` - `group__stochastic_structure` - `group__investment_temporal_block` - `group__temporal_block` Perhaps in many cases the user wants to create a specific temporal structure for all biomass nodes or all connections, nodes and units in a neighbouring country. The new class `group` could make these kind of cases easier. However, there are also cases when the user wants to create a specific structure for just one node, for example. In those cases the additional layer can decrease usability. 2. Create a new class `type` (or something similar) with fixed entities `investment`, `flow`, `on` (or something similar, corresponding to the variable names). Then, replace classes `...__investment_stochastic_structure`, `...__stochastic_structure`, `...__investment_temporal_block`, `...__temporal_block` with new classes `...__type__stochastic_structure` and `...__type__temporal_block`. However, we should somehow be able to indicate for the user which are valid combinations. This can make the structure just more complicated to understand. 3. Similar to 1. but create the `group` class to group temporal blocks and stochastic structures instead of connections, nodes and units. For that purpose, create also classes `stochastic_structure__group` and `temporal_block__group`. Replace corresponding `...temporal_block/stochastic_structure` classes with - `connection__investment_group` - `model__default_investment_group` - `model__default_group` - `model__group` - `node__investment_group` - `node__group` - `unit__investment_group` - `unit__group` 4. Replace `temporal_block` and `stochastic_structure` classes totally with `stochastotemporal_structure` class, perhaps with a parameter `type` and parameter value list [`temporal data`, `stochastic data`]. 5. Use Map parameters instead of temporal and stochastic classes. For example, create a `model` class parameter `temporal_blocks` with contents block01, block02, block03. Where to give the necessary data for the blocks? In the same (multi-dimensional/nested) Map or elsewhere? How to make sure that the user gives consistent data? Using upcoming features of Toolbox? 6. Could/should we change `stochastic_scenario` and `stochastic_structure__stochastic_scenario` classes somehow? **Ideas for other classes** - To/from nodes: Should we change the classes with `to_node` and `from_node`? Some would prefer having `direction__node`, that is, more dimensions in the classes. Some would prefer to use `input` and `output` instead of `to` and `from`. - User constraints: Is there a way to clarify the user constraint classes? Can they somehow be combined with the `group` class (that groups connections, nodes and units or temporal blocks and stochastic structures; options 1. and 3. above)? - Reports and outputs: In the input database, the additional layer of `report` in defining how the outputs are written can feel cumbersome. Is there a way to use entity groups for that? Maybe not? - Remove `unit__commodity` class?

[clizbe]

@nhniina Can you please update the description with the outcomes of the discussion? That'll make it more clear what the programmer needs to do. :)

[nhniina]

The decision was to first implement a database upgrade tool here https://github.com/spine-tools/SpineOptPlugin/tree/datastructure_update, and once that works, implement the required changes in SpineOpt code. Most of the planned updates are already implemented in the database upgrade tool. The remaining conversions are related to unit flow ratios (efficiencies) and connections.