SpineOpt Documentation #280

Closed spine-o-bot closed 11 months ago

In GitLab by @mihlema on Apr 23, 2020, 09:58

As this one is important for new users joining the Spine Group but also for us, we should probably think about how we want to tackle and set up SpineOpt documentation. My first idea is that we possibly want multiple documentation instances:

As this was getting lost in the comment section, I'm pasting the plan here:

[x] # Introduction (@jkiviluo, at Dennis, or others)
- [x] ## Table of contents?
[x] # Getting started (@jkiviluo, at Dennis)
- [x] # Installation
- [x] # Running an Optimization
- [x] # Creating Your Own Model
[x] # Basic concepts / Exhaustive reference? (@DillonJ)
- [x] ## Probably needs a lot of real-time revisions from everyone
[x] # Advanced concepts/modelling?
- [x] ## Stochastic stuff (@Tasqu)
- [x] ## Investments (@DillonJ, @KristofPhillips95 & @ Tim can assist?)
- [x] ## powerflow (@DillonJ, @KristofPhillips95, @iasonask ?)
- [x] ## decomposition (@DillonJ)
- [x] ##Temporal structure (@mihlema , @KristofPhillips95)
- [x] ## User constraint creation, etc. (@DillonJ )
- [x] ## Unit commitment (@mihlema, @KristofPhillips95)
- [x] ## Ramps and Reserves (@mihlema, @KristofPhillips95)
- [x] ## Pressure driven gas transfer (@mihlema)
[x] # Mathematical formulation (@mihlema , @TimMertens , @KristofPhillips95)
- [x] ## Objective function (@TimMertens)
  - [x] ### Fuel costs
  - [x] ### Etc…
- [x] ## Constraints
  - [x] ### Balance constraint (@mihlema)
  - [x] ### Unit operation (@mihlema)
    - [x] #### Static constraint
      - [x] ##### Conversion constraint / limiting flow shares inprocess / relationship in process. TODO: add max/min ratio
      - [x] ##### Unit capacity bounds
    - [x] #### Dynamic constraints
      - [x] ##### commitment constraints (@mihlema, @KristofPhillips95 might revise)
      - [x] ##### Ramping constraints (@mihlema, @KristofPhillips95 might revise)
      - [x] ##### Reserve constraints (@mihlema, @KristofPhillips95 might revise)
      - [x] ##### Bounds on commodity flows
  - [X] ### Network constraints (@DillonJ, @iasonask , @manuelma ? @KristofPhillips95 can assist)
    - [X ] #### Network representation (basic ratios)
    - [ X] #### Connection capacity bounds
    - [X] #### PTDF
    - [X] #### Nodebased DC lossless powerflow
    - [x] #### Gasnetwork constraints (@mihlema)
  - [x] ### Capacity transfer (investment?) (@DillonJ, @TimMertens & @KristofPhillips95 can assist)
    - [x] #### Early retirement of capacity
  - [x] ### User constraints (@DillonJ, maybe @nnhjy can assist, as he was using it already?) Explain how the model structure controls these!
[ ] # API Reference (@manuelma)
- [ ] ## Auto-generated stuff from the source code docstrings.
[ ] # Developer guidelines (t.b.d.)? (can come later)
- [ ] ## How to write variables, constraints, objectives
- [ ] ## Recommendations for debugging (?)
- [ ] ## (Move documentation from stochastic here)
- [ ] ## ...(Ideas?)
[x] # Contribution guidelines (t.b.d.)?(can come later) Other:
[ ] Address feedback from summer student (@mihlema, @KristofPhillips95 ?)
[ ] Address feedback from summer student (@DillonJ I think you also had a summer student?)
[ ] Address getting started guide suggestion #377 (@jkiviluo)
[ ] Add entry on unit operating modes to docs #89 (@manuelma)

General structure for our advanced concept sections.

Title & few basic lines of introduction, motivation
Key concepts (important,essential objects,relationships,parameters)
How to set it up? (step-by-step & examples)
Reference to constraints

Style guide for the mathematical formulation

Parameters: p_{parametername}(ind,di,ces); e.g. p{unit_capacity}(u,n,d,s,t)
Variables: v_{variable_name}(ind,di,ces); see Sets.md for reference

Tips and tricks:

Align tables:
- :--- = left align
- ---: = right align
- :---: = centered
Duplicate Headers
- If you want to give your section a name, that has already been used before you can assign a custom id for it:
  - # [Header](@id my_custom_header_name)
  - [Custom Header](@ref my_custom_header_name)
Referencing to a chapter with a different name than the linked word
- If you want to create a link to a chapter with a different name than the word in brakes, you can proceed similarly as for duplicate headers:
  - # Variables
  - [unit\_flow](@ref Variables)

In GitLab by @mihlema on Apr 23, 2020, 10:04

changed the description

In GitLab by @mihlema on May 14, 2020, 08:47

Edit: As the current model development is moving quickly, it is not desirable to provide a comprehensive documentation, that needs to be continuously be updated. To provideat least some kind of documentation, we will add descriptions to parameter definitions in order to provide guidance for first time users. Furthermore, there should be an intermediate how-to page (e.g. wiki page, SpineModel/examples) referring to relevant issues/other wikipages in order to help new users.

In GitLab by @Tasqu on Jun 12, 2020, 09:03

Well to get started, we might want to decide on the format for the User's Guide and set up a template and a rough layout at least. @manuelma do you have any suggestions? Can Documenter.jl be used for the User's Guide, or is it strictly for automatically generating documentation from the source code? On the toolbox side, the user's guide is made using Sphinx, and there seems to be some Julia side for it too: https://bastikr.github.io/sphinx-julia/, if Documenter.jl cannot be used for this.

In GitLab by @Tasqu on Jun 12, 2020, 09:10

Seems like there's already a foundation in place for the Documenter.jl in the /docs/ folder, and I just haven't paid it any attention before... Whoops.

In GitLab by @manuelma on Jun 12, 2020, 10:40

Documenter can do all that, it's a very complete package. It can also deploy docs on our gh pages and I'm trying to get that to work as well (so whenever we push to dev, docs are autogenerated and hosted.)

To effectively write the user guide, I think we need to chose a style and it will be easy. Personally I'm a fan of the "to achieve A, do B" style (as opposed to "you can achieve A by doing B" and "A is achieved by doing B" and other variants. Apart from that, I was thinking we can have the following sections:

1. Overview (how to call run_spinopt and stuff)
1. Generating variables (one subsection for each variable)
  - 2.1. Unit flow variables (e.g.: To generate unit flow variables, specify the following relationships: blablabla)
1. Adding objective terms
1. Adding constraints
1. Setting up the rolling structure
1. Model extension

That's a crude draft, but you get the idea. I don't know if something like that could work.

One more thought, maybe in the overview section or in an appendix, we can lay out the entire model using @DillonJ 's style for describing the investments. That is, listing object classes, relationship classes, parameters, etc.

In GitLab by @Tasqu on Jun 12, 2020, 11:17

mentioned in commit 6949a411b93a9001818a474e73f19e7679374cba

In GitLab by @Tasqu on Jun 12, 2020, 12:15

mentioned in commit 4931f6e1b84309059722574e0e16c63e7947719f

In GitLab by @jkiviluo on Jun 12, 2020, 13:50

changed title from Spine{-Model-} Documentation to Spine{+Opt+} Documentation

In GitLab by @jkiviluo on Jun 12, 2020, 13:50

changed the description

In GitLab by @jkiviluo on Jun 12, 2020, 13:54

changed the description

In GitLab by @manuelma on Jun 13, 2020, 20:46

Just saw commits 6949a411 and 4931f6e1, great!

Thinking more into it, we have a lot of functions just to improve readability (like all those _contraint_indices functions that the 'real' constraint_indices function calls to look good), and I'm not sure we need to document those. In principle, we should have very good docstrings for the functions and types and macros we export. At the moment, that would be run_spineopt, rerun_spineopt, @fetch (for some reason), and {variable}_indices.

So we need to think what it is that we want to export. Maybe generate_temporal_structure, generate_stochastic_structure, roll_temporal_structure are good candidates - I can imagine other packages wanting to call those. Do we want to keep exporting @fetch?

Edit: regardless, having dosctrings for our internal functions is really useful for development, even if we never export them.

In GitLab by @manuelma on Jun 19, 2020, 15:18

I think it's also important to have a consistent style in docstrings. For example, we are now combining the descriptive form with the imperative form (where the latter is preferred by julia). Also, docstrings shouldn't start saying "This function...", but go directly to the point.

I propose we follow the standard julia style describred here. They have some really nice examples in there we can borrow.

In GitLab by @Tasqu on Jul 9, 2020, 13:05

mentioned in commit fdf857f4d98d43f4ebe0d0f9b0858c23fb502c25

In GitLab by @Tasqu on Jul 10, 2020, 08:21

mentioned in commit 6fcf89a93a853e94fb6f71ef530c36dd7500f7e9

In GitLab by @Tasqu on Jul 10, 2020, 08:40

mentioned in commit b2548d5cc9f7b1a4be253a42402be6b29ea8723d

In GitLab by @Tasqu on Jul 10, 2020, 09:41

mentioned in commit 20dcce708ee051152e45e498dcff4c3fbb07dd90

In GitLab by @Tasqu on Jul 10, 2020, 10:16

mentioned in commit 1a0e7723ceaea4b97b3c8e3dc051e8764392568a

In GitLab by @Tasqu on Jul 10, 2020, 10:28

mentioned in commit 4de788c9674834f0d01f29b73cab4e84ff726591

In GitLab by @Tasqu on Jul 10, 2020, 10:34

mentioned in commit 6d503a36e7cc8d22773111e838b8a6b3aaaf23e7

In GitLab by @mihlema on Jul 14, 2020, 09:44

From the issue description above, in particular the following aspect is missing

Mathematical formulation (style similar to e.g. TIMES docu):

.pdf document describing the mathematical problem as whole

(updating model formulation repo)

Below is a possible outline for our topic oriented description. Feedback is very welcome - some parts don't seem to flow logically yet. One might also consider to add unit,connection,node explanations to "1. Model structure". This is e.g. how it's done in the backbone publication. However, if we already have this in our user guide, it can be short here.

Model structure
1. Temporal structure
2. Stochastic structure
Model equations
1. Objective function
  1. Fuel costs
  2. Etc…
2. Constraints
  1. Balance constraint
  2. Unit operation
    1. Static constraint
      1. Conversion constraint / limiting flow shares inprocess / relationship in process
      2. Unit capacity bounds
    2. Dynamic constraints
      1. commitment constraints
      2. Ramping constraints
    3. Reserve constraints
    4. Bounds on commodity flows
  3. Network constraints
    1. Network representation
    2. Connection capacity bounds
  4. Balance constraint
  5. Capacity transfer (investment?)
    1. Early retirement of capacity
  6. User constraints
Examples

In GitLab by @mihlema on Jul 14, 2020, 09:54

changed the description

In GitLab by @DillonJ on Jul 14, 2020, 09:57

Minor point is that a format other than PDF that can be edited more easily would be great

In GitLab by @mihlema on Jul 14, 2020, 10:41

changed the description

In GitLab by @Tasqu on Jul 15, 2020, 12:16

mentioned in commit d63bb5b1433a9180b92752bbae54be08f99c065f

In GitLab by @Tasqu on Jul 16, 2020, 04:31

mentioned in commit 6f7e928f43ed2007b8f9e85139dde647a777d39c

In GitLab by @Tasqu on Nov 26, 2020, 12:35

Here's the link to the slides we used during the 2020 Virtual consortium meeting. I thought it might be useful to have these here for future reference.

In GitLab by @DillonJ on Nov 26, 2020, 13:05

Some thoughts I had following the meeting:

It would be super nice to make a link between :
- the reference documentation
- the constraint equation julia code
- the constraint equation mathematical formulation

With this in mind, some things that would be nice to do:

For each parameter, find (programatically) what constraints it is part of
Auto-generate links to the relevant julia constraint files and/or mathematical formulation reference which each parameter or variable appears in

Would something like this be possible?

In GitLab by @Tasqu on Nov 27, 2020, 07:00

assigned to @Tasqu

In GitLab by @Tasqu on Nov 27, 2020, 07:03

Just to let you guys know, I thought I'd revise the current documentation to better reflect what we discussed yesterday. Avoiding duplicate work, and all that, in case anyone was thinking the same thing.

In GitLab by @Tasqu on Nov 27, 2020, 07:23

mentioned in commit 4582970ba6bf15eab60cd9537acef161398c3896

In GitLab by @manuelma on Nov 27, 2020, 07:47

I'm not sure, it's certainly possible and fun to implement but feels a bit convoluted. Isn't it easier to create those links manually?

In GitLab by @Tasqu on Nov 27, 2020, 11:25

mentioned in commit 85868120828a091a8d65821b2b97cfe59058410f

In GitLab by @Tasqu on Nov 27, 2020, 11:28

So I did a little restructuring, hopefully making the documentation a little bit easier to understand and navigate. I also separated some of the contents into multiple files, as some of the .md files were getting pretty long.

In GitLab by @Tasqu on Nov 27, 2020, 11:28

unassigned @Tasqu

In GitLab by @DillonJ on Nov 30, 2020, 10:24

I was thinking... it would be nice to write some code that generates a sort of reference index based on the contents of the template JSON. It would provide a nice MD view, in table form, lets say, of the data structure and parameters with brief descriptions and links to their fuller descriptions which we would need to embed in the descriptions.

In GitLab by @manuelma on Nov 30, 2020, 10:28

There's the function write_system_components_file in src/util/write_information_files.jl that maybe does or starts doing that, just in case you haven't seen it yet.

In GitLab by @mihlema on Dec 1, 2020, 23:59

For reference some other open models and their docs:

https://calliope.readthedocs.io/en/stable/ (~similar to ours)\ http://www.balmorel.com/index.php/the-balmorel-concept https://pypsa.readthedocs.io/en/latest/index.html https://osemosys.readthedocs.io/en/latest/?badge=latest https://oemof.readthedocs.io/en/v0.0.5/overview.html https://urbs.readthedocs.io/en/latest/Users_guide.html

In GitLab by @Tasqu on Jan 5, 2021, 10:42

mentioned in commit d119ce03c237edf22b9607772146131ec4c1d8f1

In GitLab by @Tasqu on Jan 7, 2021, 06:28

mentioned in commit eab442add755dd27068e84d9696b0f3b05c85297

In GitLab by @Tasqu on Jan 7, 2021, 06:28

mentioned in commit e7a7e3cca817136847069769374c1f154d0bdbc9

In GitLab by @Tasqu on Jan 7, 2021, 06:43

mentioned in merge request !57

In GitLab by @Tasqu on Jan 7, 2021, 06:46

I made a prototype for a new way of automatically generating the "concept reference" portion of the documentation based on the spineopt_template.json and separate description <name>.md files in MR !57. Let me know if you think it's an ok approach, or if I'm missing something.

In GitLab by @Tasqu on Jan 7, 2021, 07:29

mentioned in commit cb303b5e21d85d2bb5b12daaf930fb182dce3994

In GitLab by @Tasqu on Jan 7, 2021, 07:39

mentioned in commit 3b1e2605f7472a281e3db54043a889c05a74366c

In GitLab by @Tasqu on Jan 7, 2021, 11:16

mentioned in commit fe8bd084ae070dc638480a8d10ceb5183aa08f73

In GitLab by @Tasqu on Jan 7, 2021, 11:23

The prototype in !57 now also processes most of the important information in the spineopt_template.json automatically into the Concept Reference chapter of the documentation, with working cross-references.

In GitLab by @Tasqu on Jan 8, 2021, 10:07

mentioned in commit 14f965c2a041cc84d32304af9614c6f23eccbc1c

In GitLab by @Tasqu on Jan 8, 2021, 13:05

mentioned in commit 16bc6534f4ebb501123926508fb154a6ee270b63

In GitLab by @Tasqu on Jan 8, 2021, 13:09

mentioned in commit 95f9060f49503f460b489101bc05367078ff23d6

In GitLab by @Tasqu on Jan 11, 2021, 10:03

mentioned in commit ec4de2897fbddd44ce0cbf7d1130a241a3cf1bf1

spine-tools / SpineOpt.jl

SpineOpt Documentation #280