Documentation Improvements

[robkooper]

From an older issue @infotroph about documentation: https://www.divio.com/blog/documentation/

More context: This piece makes a strongly-argued case that there are four distinct types of software documentation, and that all well-documented projects need to have all four of them as explicitly separate sections:

• tutorials, for teaching beginners what your tool does using step-by-step examples that are guaranteed to work exactly as described every time

• how-tos, the cookbook section where users can go to answer questions of the form "How do I....?", containing only the details they need for a given question

• reference, for the man-page details on how to call things, what protocols they speak, and what values they return

• discussion, where you explain why things work they way they do, provide background information, advise on good vs bad practices, and otherwise give context that don't fit in the other sections.

[robkooper]

Looking at the documentation https://pecanproject.github.io/pecan-documentation/develop/ the menu on the left is large and seems not to match what we have. Do we want to have 4 different books that match better with the above, another would be the following 3 books

Overview

• what is PEcAn
• who is it for

User Guilde

• how to use PEcAn
• links to vignettes (esp if running locally)
• adding inputs/data
• using HPC
• using rstudio
• pecan.xml documentation
• list of models

Developer Guide

• git workflow
• compiling pecan
• adding new models
• how to debug

Admin Guide

• different installs
• docker -- what is docker -- install using docker -- troubleshoot docker
• VM -- virtualbox -- how to install -- troubleshooting
• Native -- what OS we support

[robkooper]

would like feedback from a technical writer as part of GSOD.

[theadityasam]

Hey everyone, this is Aditya Samantaray from IIIT Bhubaneswar, India.

I had been discussing the exact same issue with @istfer right after the GSOC proposal period, as I, being new to Pecan, felt the lack of a "beginner's guide to Pecan and contributing to Pecan" in the docs.

I was planning on opening a new issue after coming up with a proper plan for new documentation that I would be able to work on during the quarantine period, but had gotten stuck with some of my personal courses and didn't get much time to think about it. Now that I've seen this issue, I believe that documentation restructure is what this community desires too.

Getting all of that out of the way, I have few things to add to this issue that I believe might be helpful.

• Changing the documentation structure: I believe that currently, the documentation is jumbled and very hard to navigate. I agree with the documentation structure proposed by @robkooper above, which will keep the docs clean and structures. I would like to add to that structure, a fifth bookdown - a bookdown for the theory of the models, their input and output arguments, their implementations and the code references to the github page. This would be great for researchers as well as beginners who need and insight on what all PEcAn has to offer and it would also be helpful to the existing users, acting as a one stop reference guide for ecosystem modelling in general.

• Changing the build system for documents: I might be wrong about this, but currently the bookdowns reside in book_source and manually the deploy.sh is being run to upload and update the documentation. Rather, it would be better to move the bookdowns to the master of pecan-documentation and leave the deployment to Travis(something like this).

• Working on the content of the docs: There are some parts of the documentation that are incomplete(for eg: the models page). This will be the longest and most important part of this issue. PEcAn is a big project and documenting each element of PEcAn requires days of proof reading and inputs from the various contributors on the elements. Although for the major part, documentation has already been written, it needs to be restructured and rearranged. In fact this section alone summons the need of a separate GSOD for it wherein new Rmds will be written and existing ones will be edited.

So, tldr; The existing documentation needs to be restructured (references as suggested by @robkooper-divio blog, django, divio), The book_source should be moved to master of pecan-documentation and hosted in the gh-pages branch, automatically deployed by Travis upon every commit-something like this, Missing details need to be filled, new guides need to be written (new section for theory of ecosystem modelling and models would be cool) - best to be left for a technical writer during GSOD.

I can work on the first two parts of what I proposed, i.e. basically creating a skeleton for the documentation and porting the old documentation into it, along with minor additions of Travis triggers as mentioned above.

[infotroph]

leave the deployment to Travis

We actually do this already! But in a way that doesn't make much sense, so don't feel bad about having missed it: The bookdown source is compiled in the after_script step of the Travis build, which calls a series of scripts that live in at least three different directories and ultimately winds up running documentation/tutorials/deploy.sh to perform the deployment to the pecan-documentation repository. These scripts are long overdue for a refactor, but in the meantime the deployment does mostly work.

We have a long-simmering goal of moving most of these deployment activities to GitHub Actions instead of Travis, and it would be great to streamline the process at the same time we move it.

[robkooper]

Nice thing about github_actions is that we can now ignore certain folders when building and vice versa. So now we can have the documentation only build if there are changes and not build the main code if there are no changes in the main code.

[theadityasam]

We actually do this already!

Ohh I see, I just read the travis yaml and found the scripts. Sorry for missing it :P

We have a long-simmering goal of moving most of these deployment activities to GitHub Actions instead of Travis, and it would be great to streamline the process at the same time we move it.

Sounds like a plan. Also @infotroph, what do you think about moving the docs to the master of pecan-documentation. The bookdown can then be deployed to github pages(on gh-pages branch).

For now, let's create a new repo (let's say pecan-docs), upload the bookdown source over there, setup github actions for it, and deploy it? This shouldn't take more than a few days I believe. Then we can work on the actual contents of the Rmd's itself.

[infotroph]

what do you think about moving the docs to the master of pecan-documentation

My understanding is that keeping book_source inside the PEcAn codebase rather than in its own repo was a considered decision, but it happened before I joined the project and @mdietze or @dlebauer can speak better than I about the exact reasons.

[robkooper]

Problem with it being in its own repo that if something has changed that would require an update to the documentation it would need to happen as 2 pull requests. One against the source code and one against the documentation.

The goal of having this in a single repository is for the person reviewing to make sure that any changes to the code that warrants an update to the book code also is reflected in the source code of the book, and if not mark the pull requested as "needs imrpovements".

[ashiklom]

My understanding is that keeping book_source inside the PEcAn codebase rather than in its own repo was a considered decision

Yes, this was a very deliberate decision, and one that I think we need to stick to. Two closely related reasons: First, the reason Rob mentioned above --- it allows us to enforce documentation revisions alongside major code changes. For instance, if I decide to change how the met processing code works, this will allow admins to block my pull request until I've also updated the documentation accordingly. Second, it helps ensure that documentation and code stay in sync with each other -- i.e. if you checkout an older version of PEcAn (for whatever reason), you can also easily access the older version of the code. Similarly, it allows discussions about proposed changes to both code and related documentation to be stored in the same Pull Requests, which makes it somewhat easier to go back through the history to see what changed and why.

All that said, exactly where in the PEcAn codebase the documentation lives can certainly change. I think book_source is an unfortunate and uninformative choice of directory name, and there's lots of room to refine the file structure within there.

[theadityasam]

First, the reason Rob mentioned above --- it allows us to enforce documentation revisions alongside major code changes \

Second, it helps ensure that documentation and code stay in sync with each other

Okayy, I gettit. It should stay there then. So basically, things that can be considered objectives are - setting up github actions, restructuring and cleaning book_source, and also restructuring the documentation. Correct?

[github-actions[bot]]

This issue is stale because it has been open 365 days with no activity.

[allgandalf]

More context: This piece makes a strongly-argued case that there are four distinct types of software documentation, and that all well-documented projects need to have all four of them as explicitly separate sections:

I agree with @robkooper that we need clear documentation to onboard more people into PEcAn community, if season of docs is not in discussion then we should give a thought to include this issue in GSoC'24 as one of the potential projects :), will like to discuss this further if all are onboard with the Idea

[Sweetdevil144]

Do we want to have 4 different books that match better with the above, another would be the following 3 books

I would say we move with the 3 book Strategy. However, I can propose this as a 4 book strategy as suggested in Issue description. Here's a succinct reorganization book strategy for the PEcAn documentation:

Overview Book

• Introduction: What PEcAn is and its ecosystem.
• Audience: Who it's for, skill level needed.
• Concepts: Core ideas, architecture, data flow.

User Guide Book

• Getting Started: Installation and setup instructions.
• Using PEcAn: Step-by-step guides for common tasks, link to vignettes.
• Advanced Topics: HPC usage, pecan.xml deep dive, model listings.

Developer Guide Book

• PEcAn Development: Git workflow, environment setup.
• Model Integration: Guidelines for adding new models, debugging tips.
• Code Contribution: Best practices, review process.

Administrator Guide Book

• Installation Guides: Docker, VM, and native installation steps.
• Maintenance: Updating PEcAn, database synchronization.
• Troubleshooting: Common issues and their solutions.

NOTE : What I provided here is a layman overview of how things can be covered broadly including all the aspects of the current documentations ad hoc to DIVIO DOCUMENTATION SYSTEM

[Sweetdevil144]

What are your opinions regarding this @RohanSasne ?

[mdietze]

First, by "book" I assume we mean major subsections of the documentation, not distinct documents, as cross-referencing and maintaining the documentation is easier in the former

Second, I don't see the "Overview" as a major section. If it takes pages of general description and conceptualization before we get to the first "Hello World" style tutorial, people aren't going to stick around. Per Rob's original post, I think it makes sense have the first two sections be "Tutorials" and "User guide" -- show people how things work first, with enough conceptual scaffold that the build the right mental image of how PEcAn works, and then follow that up with the gory details. So move up the current "Using PEcan" subsection and demote the Audience and Concepts

Also not that Installation currently shows up in two places (User Guide and Admin) -- that either needs to be consolidated or a clearer conceptual split as to what sort of installation goes in each half. I tend to think Installation needs to be in the Users guide but Continuous Integration needs to be distinct part of Dev or Admin -- Github Actions drives new developers nuts and as an Admin keeping it working drives me nuts, but the alternative is a vastly less reliable system.

[Sweetdevil144]

First, by "book" I assume we mean major subsections of the documentation, not distinct documents, as cross-referencing and maintaining the documentation is easier in the former

Absolutely @mdietze. Similar to present documentations, we can divide the Present documentation into 4 or 3 'Heading' subsections/books as decided by @robkooper later on, with detailed explaination of the present sebsection in each. Furthermore, the subsections will contain sub-sections which will be decided.

Second, I don't see the "Overview" as a major section. If it takes pages of general description and conceptualization before we get to the first "Hello World" ....... ..... drives new developers nuts and as an Admin keeping it working drives me nuts, but the alternative is a vastly less reliable system.

Thank you for correction for the documentation structure. Based on your suggestions, we will streamline the 'Overview' section for brevity and immediate engagement, prioritize 'Tutorials' and 'User Guides' for early practical insights, and ensure clear distinction and consolidation in the 'Developer' and 'Administrator' guides, particularly regarding installation content. Your insights are instrumental in enhancing the clarity and utility of our documentation.