search

spine-tools / SpineOpt.jl

A highly adaptable modelling framework for multi-energy systems
https://www.tools-for-energy-system-modelling.org/
GNU General Public License v3.0
55 stars 13 forks source link

SpineOpt Documentation #280

Closed spine-o-bot closed 1 year ago

spine-o-bot commented 3 years ago

In GitLab by @mihlema on Apr 23, 2020, 09:58

As this one is important for new users joining the Spine Group but also for us, we should probably think about how we want to tackle and set up SpineOpt documentation. My first idea is that we possibly want multiple documentation instances:

As this was getting lost in the comment section, I'm pasting the plan here:

General structure for our advanced concept sections.

Style guide for the mathematical formulation

Tips and tricks:

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 11, 2021, 12:20

mentioned in commit 8b044b05b65a25bd0fc762035e0e0116e0ba9451

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 11, 2021, 12:20

mentioned in commit 7dc28d50eeb1a4757bb4f43a6c9346559217eae4

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 12, 2021, 06:16

mentioned in commit 20a73daea5d2c6286356058e0479c38aa4325ca9

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 12, 2021, 08:52

mentioned in commit c1da1696454e8ae65d81197bfe64e3613762c250

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 12, 2021, 09:39

mentioned in commit 9b981f136e9d78cf25174fcd745479cf9ce297a0

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 12, 2021, 09:40

mentioned in commit 631674e7bbf93f64b95099e5379b8bc1e84f334c

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 12, 2021, 10:18

mentioned in commit 5d6e3612641b00c1cabdca14b7a417b8be3a8d80

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 12, 2021, 10:53

mentioned in commit 287d2791b2b135e9e3dc56c438ff81cc163318ac

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 12, 2021, 11:12

mentioned in commit 4beb7c81d4e17b642a45b32a091407da757f0388

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 13, 2021, 09:38

mentioned in commit c9815a13841680d492cb76f8c6d710064109dfcf

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 13, 2021, 10:13

mentioned in commit a4a5ba8d5362538af9ad81a0830557998cc3bf0a

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 14, 2021, 11:32

mentioned in commit b34de8ec6c58e9d1da957fc4a5230aa0c6fa07db

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 18, 2021, 09:14

mentioned in commit 47601a1eb08fca70012c8fc46e83e9803b9cea6f

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 18, 2021, 09:14

mentioned in commit f6f974ebca49a86df7335ff7faed1798039bd4e3

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 18, 2021, 09:52

mentioned in commit 47697c473dfef578cd57cf6026dcf73549f6fd1e

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 18, 2021, 10:53

mentioned in commit 4708d811d8b9d7683c547ce0a4ccb1eb909f7a9e

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 18, 2021, 11:20

mentioned in commit f271167b3b9d529db58ee5a5ccaf345d5032d96e

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 18, 2021, 11:34

mentioned in commit cea5b8a965b22be586244ac3352973f4ac6f385f

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 18, 2021, 11:57

mentioned in commit 9b7942a674700e00ce608cb9e5c49b496c47aa76

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 20, 2021, 13:50

mentioned in commit a302cb041212f54431661899d255a97e136fc73e

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 26, 2021, 09:54

mentioned in commit 8819fb1d639654c116b40939e9b5c1be9dcf4ad0

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 28, 2021, 07:44

mentioned in commit 169f823c2379acca2291a0972eb6ee9e13f228e4

spine-o-bot commented 3 years ago

In GitLab by @Tasqu on Jan 28, 2021, 07:46

I added illustrations to the Stochastic Framework section of the documentation, but couldn't find a way to resize the images using the native julia flavored markdown. I used raw html to bypass the issue for now, but if anyone happens to know a cleaner syntax for this, please let me know.

mihlema commented 3 years ago

See also: https://docs.google.com/presentation/d/1Rk6N-geW65LgaSMqrKx_WdrEAIJoYY1Ahc06wv29o1g/edit#slide=id.ga346e9ea10_0_70

mihlema commented 3 years ago

Given the upcomind miletsone, I added the documentation plan here, added some more detail and tried to add people to specific topics (we can discuss this tomorrow):

dennis commented 3 years ago

I am sure you don't want me to do any documentation :) So it's probably not @dennis you wish to tag :)

mihlema commented 3 years ago

I am so sorry, @dennis.Thanks for pointing this out!

DillonJ commented 3 years ago

I had an initial stab and pushed something on investments, but it doesn't cover decomposition yet.

Here's a question... I'm not 100% sure what needs to be done on a practical level for each of the to-do items above. For investments it was easy because there was an .md file in docs\advanced_concepts.

Are we saying that (for example) :

Or something else? Perhaps someone could have a go at creating this high-level structure populating files with #TODO as required?

Tasqu commented 3 years ago

I checked the Introduction and Table of Contents, as there are already (brief) sections about those in the documentation. I suppose the Introduction could still be fleshed out to give potential users a better overview what SpineOpt does, though.

As for the Table of Contents, I don't think there's a need for such a thing, at least in a traditional sense. There's already a list of the different chapters etc. in the sidebar that's automatically generated, so I don't think we need to include another list manually. However, I think some summary of the contents of the documentation can be helpful for new users, and there's already a bit of that on the Introduction page of the documentation.

Tasqu commented 3 years ago

I had an initial stab and pushed something on investments, but it doesn't cover decomposition yet.

Here's a question... I'm not 100% sure what needs to be done on a practical level for each of the to-do items above. For investments it was easy because there was an .md file in docs\advanced_concepts.

Are we saying that (for example) :

  • level # are a so-named md files in \docs with a sub-folder if there are children
  • level ## are so-named md files in \docs#level_1_name
  • level ### are sections within a level ## md file

Or something else? Perhaps someone could have a go at creating this high-level structure populating files with #TODO as required?

@DillonJ

Basically, the docs/make.jl defines how the documentation is structured. At the moment, There is the introduction index.md and four "chapters" Getting Started, Concept Reference, Mathematical Formulation, and Advanced Concepts, which contain pages on different topics. Currently, I've organized the files similar to how they are organized in the documentation: index.md in the root docs/ folder, and a sub-folder for each "chapter" containing all .md files in that "chapter". The docs/concept_reference is the only odd one atm, as it houses some automatically generated files and the description .md files for each individual concept.

In general, each page is it's own .md file, and all the pages are shown in the side-bar of the documentation based on the title assigned to them in make.jl under the "chapter" they have been allocated under (if any). So if you want to add a new page under a "chapter", make a new .md file in the correct sub-folder, and include it in docs/make.jl under whichever chapter it relates to with whatever title you want.

The structure of the documentation has almost nothing to do with the # levels in markdown! So far, I've started each page with its #Title followed by ##Subtitles and ###Subsubtitles as necessary. It seems that the current depth of our documentation generation opens a list of all ##Subtitles in a page in the side-bar when that page is selected.

I can definitely add page templates for the remaining Advanced Concepts if you want, but I'd prefer if the people who are actually going to write the documentation did that themselves. You probably know how to properly structure your portion of the documentation better than I ever would.

Tasqu commented 3 years ago

Also, about the Documenter.jl module:

I fixed some of the references the investment section by @DillonJ broke, but I'd prefer if I didn't have to do it every time the documentation is updated.

Tasqu commented 3 years ago

Ok, I've cleaned up the current master so that building the documentation only throws the following warning (that I don't understand):

┌ Warning: Documenter could not auto-detect the building environment Skipping deployment.
└ @ Documenter C:\Users\TRTOPI\.julia\packages\Documenter\PLD7m\src\deployconfig.jl:41

Note that automatically building the Concept Reference (before building the actual documentation) still throws a whole lot of warnings. This is by design to warn the user of any missing description files (which there are currently a ton of).

Tasqu commented 3 years ago

Note to self @Tasqu: Check the gh pages branch for the documentation and try to see if something is broken. Try including the temporary files in the repository to see if that fixes it.

Tasqu commented 3 years ago

Ok, as far as I can tell, the automatically generated Concept Reference should now work in the online documentation. It seems that for whatever reason, the underscore in the temporary file names e.g. _object_classes.md caused the browser to not find the correct .html file, even though it was generated correctly.

However, I noticed that the images aren't being displayed in the online docs either, so I guess I'll look into that while I'm at it.

Tasqu commented 3 years ago

As far as the images go, this has turned out to be a weird issue with two main problems:

  1. Building the docs locally results in different output than when the docs are built for gh-pages. For some reason, the gh-pages build always has an extra layer of folders, e.g. local advanced_concepts/stochastic_framework.html becomes advanced_concepts/stochastic_framework/index.html, which messes with relative paths to try and find the figs/ folder. This makes it so that the relative source paths for the figures in the documentation cannot point at a valid location both locally and in gh-pages.

  2. The symbolic link latest -> dev doesn't work for image source paths. The documentation at https://spine-project.github.io/SpineOpt.jl/latest/ seems to be mapped to point at https://spine-project.github.io/SpineOpt.jl/dev/, but the image source paths don't seem to work in the former. Currently, I've set up the figures in a way that they should display in the dev documentation online, but I don't know how to fix them for latest.

@manuelma if you have any idea what's causing these problems, I'd appreciate the help.

Tasqu commented 3 years ago

Ok, the different formatting is caused by line 43 of make.jl:

format=Documenter.HTML(prettyurls=get(ENV, "CI", nothing) == "true"),

which formats the output different depending on where it's being built. By removing this line, Documenter.jl defaults to prettyurls=false, which is how the online documentation is built.

Unfortunately, the local documentation is more convenient with prettyurls=true, so I don't think disabling it is a great option. Ideally, we'd have to figure out some way to find the images regardless of the folder structure of the build, but I don't know if that's possible.

Tasqu commented 3 years ago

... and nevermind the point 2 of the above. It seems that the online documentation simply needed some time to sort out latest vs dev, and it seems to display the images correctly now.

mihlema commented 3 years ago

With respect to our last call, I think it'd be good if we can loosely agree on a general structure for our advanced concept sections. First idea, loosely based on the stochastic structure documentation:

  1. Title & few basic lines of introduction, motivation
  2. Key concepts (important,essential objects,relationships,parameters)
  3. How to set it up? (step-by-step & examples)
  4. Reference to constraints
mihlema commented 3 years ago

Regarding the mathematical formulation. I thought it would be nice to add a section Sets prior to the Variables, Constraints, etc. Mainly, I wanted to introduce sets we use in the code here, e.g. units_on__indices. But there is also going to be some overlap between the Concept Reference and the mathematical sets in the constraints, if we use relationships or parameters somewhere in the code (e.g. indices(max_ratio_out_in) or node__node).

Maybe we'd need a section somewhere, explaining what SpineInterface (and preprocessing) does, and giving a generic explanation for the indices(parameter;...), relationship(), object function. I am not sure where this chunk of Documentation should live, maybe in the concept reference? Then the Sets would only hold sets created within spineopt, e.g. variable sets, timeslice relationships and stochastic path indices. Alternatively, we could also cover these special sets indices (variables, temporal etc.) in the API reference documentation.

Tasqu commented 3 years ago

Maybe we'd need a section somewhere, explaining what SpineInterface (and preprocessing) does, and giving a generic explanation for the indices(parameter;...), relationship(), object function. I am not sure where this chunk of Documentation should live, maybe in the concept reference? Then the Sets would only hold sets created within spineopt, e.g. variable sets, timeslice relationships and stochastic path indices. Alternatively, we could also cover these special sets indices (variables, temporal etc.) in the API reference documentation.

  1. As far as SpineInterface functionality like calling database objects, relationships, and parameters, I feel like they should be documented in the SpineInterface repository. I don't know if there is one, though, or if we're planning on making one at any point.

  2. I suppose the index functions could be important enough to merit their own section, and I tend to agree that the Concept reference feels like the appropriate "chapter" to include those in. However, since they are a purely SpineOpt creation, there's no source that can be used for automatically generating a list of them like with the database-related concepts. Any "Indices" section would have to be made completely by hand, as far as I can tell, or rely heavily on the docstrings in the code.

@mihlema

Tasqu commented 3 years ago

I merged my improved concept reference auto-generation into master, and as far as I can tell it should be working properly. The online documentation is a bit wonky at the moment (you need to visit other sections of the documentation before you can access the concept reference from the sidebar), but I hope it will resolve itself in a bit. The online documentation seems to always take it's time to figure itself out, even when the gh-pages branch gets properly updated.

Tasqu commented 3 years ago

... And now it seems to be working properly. Figures.

DillonJ commented 3 years ago

By the way - did we ever discuss US vs British English? My preference is British - since we have few Americans involved and this is the language of the vast majority of native speakers in the project (no offense to our dear American friends @claytonpbarrows) :-)

What this would mean, mostly, is to use s instead of z everywhere :-) (optimise, penalise)

However, perhaps there are academic standards / community conventions that are relevant? Thoughts?

manuelma commented 3 years ago

I'm going with British myself, but to me, the only difference is s vs z. I don't mind either way as long as we're consistent of course.

Tasqu commented 3 years ago

I've apparently been using American spelling most of the time (I just like the letter z because we Finns don't get to use it enough), but I can try to change that.

DillonJ commented 3 years ago

OK, sounds like we can go with British then. For consistency, I guess we can correct things as we come across them. It might also be helpful to consider changing the spell check language of your browser accordingly.

DillonJ commented 3 years ago

I'm wondering if we want to update the auto documenter code to enumerate the possible values for parameter lists. This would be nice so it picks up any changes if we make them.

Edit: it could also list the parameters it applies to

Tasqu commented 3 years ago

@DillonJ as long as that information is contained in the spineopt_template.json, it's definitely possible.

mihlema commented 3 years ago

@DillonJ just thinking about the developers guideline here. If I remember correctly you had already some idea on how to set this up (with a flow chart?). Did you potentially already draw this out somewhere?

mihlema commented 3 years ago

Taking a note on last weeks discussion:

@CIODWYER, as you pointed out some problems here, do you think we'd need some more description here/would you be willing to add missing pieces?