search

spine-tools / SpineOpt.jl

A highly adaptable modelling framework for multi-energy systems
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
53 stars 13 forks source link

Archetypes - initial implementation using templates #384

Closed mihlema closed 1 year ago

mihlema commented 3 years ago

As discussed before, our initial step towards supporting archetypes will be to provide .json templates that can be imported with all necessary parameter relationships defined. This issue serves as a list of envisioned archetypes and responsibilities. Note that archetypes will also need to be documented in the online documentation #280, in general (how to use) and very briefly specific (what is each archetype for). Note that I have also added "model" archetypes (in contrast to "system" archetypes, which typically correspond to the physical system)

General description on how to use archetypes @mihlema (I feel like this should be mentioned in the relevant parts of the documentation; e.g. description of pressure-driven gas transfer)

System archetypes:

Model archetypes, t.b.d.:

I'll make an initial example for one of "my" archetypes, then others can swiftly follow.

Tasqu commented 3 years ago

Just to be 100% clear here, should we try and keep the archetypes as "minimal" as possible, meaning that they area meant for importing on top of existing DB:s and each other to serve as "templates" for doing different things in SpineOpt?

E.g. that the "archetype" for a deterministic model stochastic structure would essentially just consist of a pre-made stochastic_structure, stochastic_scenario, stochastic_structure__stochastic_scenario relationships (and maybe some parameters I'm forgetting)? This could then be imported on top of the SpineOpt template to provide a ready-made stochastic structure, that would then have to be connected to the desired parts of the model. (I guess I'd have to explain somewhere how the archetype stochastic structure is then actually connected to the rest of the model, but that would be in the documentation I guess).

Or is the purpose to be more like "tutorials", where I'd also provide ready-made model, unit, node, or something to illustrate how the stochastic_structure is connected to the rest of the model via the structural relationship classes?

DillonJ commented 3 years ago

My understanding is that they would be minimal and allow importation on top of an existing dataset.

I can see the value of the second option, however, this feels more like a sample system?

Tasqu commented 3 years ago

My understanding is that they would be minimal and allow importation on top of an existing dataset.

I prefer the "minimal" type myself, don't get me wrong. I just wanted to make sure we're trying to achieve the same thing. I'm not sure where we can easily include instructions for how to connect the archetype to the rest of the system, but I guess we'll figure something out.

I should be able to whip up archetypes for e.g. a deterministic, branching and converging stochastic structures easily enough. I'd probably make them according to the example instructions in the Stochastic Framework section of the documentation, so that I could point to them from there or the other way around.

Tasqu commented 3 years ago

FYI, it seems that due to recent changes to parameter value types, exporting and importing special parameter values from .json isn't working properly. I tried making example archetypes for a couple different stochastic structures, but importing them doesn't work properly atm due to special parameter types like Duration. I've let @manuelma know about the issue, and he said he'll work on it alongside the other type-related stuff he's doing.

I'll try and let you guys know when the .json parameter values are working as intended.

Tasqu commented 3 years ago

Ok, @manuelma fixed the above issue in the .json export/import in spinedb_api, so I've got my first working drafts for the stochastic structure "archetypes" in the WIP pull request #386. Hopefully, they can serve as a basis for further discussion.

Tasqu commented 3 years ago

It seems @manuelma already made a sort of "basic model" template for a workshop at KTH basic_spineopt_template.json, so I'm guessing we can use that for the

Basic model with 1 temporal block, 1 stochastic scenario

part? Could maybe use some cleanup and documenting, but other than that, I think that's more or less what was expected? A basic template on which users can immediately start adding units, connections, nodes, etc. without having to deal with the temporal and stochastic stuff.

Tasqu commented 3 years ago

I think we need to agree where we want to store the archetypes. Currently, the SpineOpt.jl root folder has the following sub-folders:

\data - Houses the `spineopt_template.json` (and @manuelma `basic_spineopt_template.json`)
\docs - Houses the docs environment and the source code for the documentation.
\examples - Houses some old `test_systemA5.jl` files.
    \data - Houses the input data .sqlite for the `test_systemA4.jl`
\src - Contains the source code for the SpineOpt.jl module.
\test - Contains the unit testing environment and code.

I think the root \data folder is unnecessary, as it currently only (really) houses the spineopt_template.json. Also, the name is a bit misleading in my opinion, as it's not really about "data" but more about "data structure". I like the \examples folder, as it's immediately obvious what it contains, and its potentially really helpful for new users to see. However, I think we could structure things a little nicer, e.g.

\examples - We could include e.g. the `basic_spineopt_template.json` directly under here.
    \models - This is currently only for the `test_systemA5` stuff, if we want to preserve it. We can always include more example models here when needed, e.g. my stochastic test system?
        \(potential sub-folders if needed)
    \archetypes - This would be where the archetype .json files would live. Essentially a "library" for pre-made model pieces.
        \(are sub-folders needed?)

I'm still uncertain what would be the best place for the spineopt_template.json, as it's not really an "example". In my opinion, it's integral enough to SpineOpt to be included under \src if necessary, but I think it's also even more important in the sense that it needs to be referred to by the users more often than the source code.

I would maybe just include spineopt_template.json in the root folder for maximum visibility. Maybe even rename to something more descriptive, e.g. spineopt_data_structure.json? Any thoughts, @mihlema, @manuelma, @DillonJ ?

manuelma commented 3 years ago

I believe the best is to rename data to templates and host all the templates (aka archetypes) there. Typically, examples is reserved to .jl scripts that demonstrate part of the code, and src is reserved to .jl files too. It would be weird to have a JSON in any of these folders in my opinion.

I also don't like having spineopt_template.jl in the root, but that's also just my opinion.

Tasqu commented 3 years ago

Ok, I've made the example archetypes for the stochastic structures, as well as documented them under a new section in the documentation in the pull request #386. I also reorganized the folder structure according to my proposition before, but that can be changed at a later date as desired. Check it out and comment if you have opinions.

The merge request should be good to go, if you think we should go with my proposed approach. (Although the unit tests don't pass due to some changed @mihlema made to the master a couple of days ago?)

On a side note: if we're aiming to rack up a lot of archetypes eventually, it might be worth it to auto-generate the documentation similar to the concept reference. However, I don't think it's necessary at this stage.

Tasqu commented 3 years ago

I believe the best is to rename data to templates and host all the templates (aka archetypes) there. Typically, examples is reserved to .jl scripts that demonstrate part of the code, and src is reserved to .jl files too. It would be weird to have a JSON in any of these folders in my opinion.

I also don't like having spineopt_template.jl in the root, but that's also just my opinion.

I feel like we should definitely keep at least the spineopt_template.jl separate from the other archetypes, as it's way more fundamental for SpineOpt.jl (e.g. used for auto-generation of the documentation). Also, it doesn't really contain any "working example", just the basic structure.

Tasqu commented 3 years ago

I merged my stochastic archetype stuff into master and tried to organise the files according to our discussion last Wednesday. I feel it's better to include it now than to wait until it gets forgotten.

Regardless, we can always shuffle the files around later if we feel like the file structure isn't ideal.

clizbe commented 1 year ago

Closing due to stale status.

clizbe commented 1 year ago

Closing because stale.