Documentation Restructure

MagikEh commented 3 years ago

Proposal: Restructuring of the Sarracenia documentation as to make it more beginner-friendly, more structured for experienced users, and more attractive for window shoppers.

Currently the /docs/ folder is in a bit of a sorry state, there are some half finished talks, broken links, pictures scattered about from previous installments and no real overarching direction for documentation. I propose we restructure follow a 4 step process to fully restructure the documentation of the project:

General cleanup and folder structure fix.

Move things into folders using a Unix style directive with the divo documentation as a metric, ie: Given the 4 different types of documentation, each file shall do one thing and do it well in regard to the subject and style:
Also during this stage I'd like to more aptly name things (this isn't math, we can have long variable names), ie:
- Dev -> Sarra Development.rst
- Prog.rst -> Writing Plugins.rst

An initial folder structure could look something like this:


       ├── docs/
       │   ├── Tutorials/
       │       └── Quickstart.rst
       │   ├── How2Guides/
       │       ├── Install.rst
       │       ├── Upgrade.rst
       │       ├── Examples/
       │       └── Prog.rst
       │   ├── Explanation/
       │       └── Treatises (detailed technical [Writeups | Analysis | Design reports])
       │   ├── Reference/
       │       └── Man pages live here
       │   └── Contribution/
       │       ├── Templates/
       │       ├── Contribution.rst
       │       └── Dev.rst
       │   └── Localization/
       │       ├── fr/
       │       ├── de/
       │       └── sv/

Enable and setup GitHub wiki to subtree merge into /docs/ as per experimentation over at wikiTests
- Create _Sidebar.rest and _Footer.rest to manage wiki navigation.
Re-write current docs as to follow the above structure.
- Change writing style from personal to Imperative mood.
- Man pages should be cleaned up as just reference material, excess information can be moved to how to guides. (all the while following man page styles on a template) (looking at you sr_subscribe.rst
Localization/Translation using deepl
- Duplicate top level folders into docs/Localization/<code>/ and prepend all English docs with <code>
- Proof read to follow automatic translation.

Overall I believe this will help usability of the docs from all points and all types. Weather you're a novice or expert working at contributing or extracting information, it is put together in a meaningful way that is easy to navigate with many means of access (man pages, src browsing, or online wiki work).

Questions pertinent to progress.

What files can be deleted?
- We have several half finished writeups and talks in the git tree. Are they worth keeping?
- I wouldn't mind running a purge as the git tree history will always hold these files for archival.
Should bundled plugins have their own individual man pages as well?
- At the very least I'd like them to have their own reference pages.
How far diverged are the files in /docs/fr/ ?
Which branch would you like to start this process off in?
- I was thinking v3_wip was an ideal choice as it seems to be the most forward moving.
This issue may be an agglomeration of
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 58
- and then some.

petersilva commented 3 years ago

I like it! Looks cool! some thoughts:

avoid spaces in file names... would prefer if Dev.rst became SarraceniaDevelopment (or you could have an underscrore... but spaces often cause issues... best avoided.
there is a small caveat that the debian build procedures need to be updated to include the man pages in the build process. I don't think it's a problem, just when you move them... need to update the debian/ subdirectory so that it still works (test using debuild and having the packages include the man pages.)
you have localization... fr/ de/ etc ... what tree would you see under fr/ existing? in theory the entire tree would be reproduced... but perhaps adapted per language? so Tutorials --> fr/Tutoriels --> HowTo --> fr/Recettes --> Reference -> fr/Réference (not sure if that is a false friend anyways gives the idea... would the sub-dirs be in the other language, or in the English so that all the trees are the same and correspondence is more obvious but the tree is not clear to speakers of the foreign language?
we produced our first draft of most of the existing v2 French guides using deepl... it translated "data injection" into a drug reference... and many, many other howlers. The francophones that reviewed the result found it useless. It took quite a while, with lots of humans involved to get the French in v2 to a usable state. That was about three years ago. Not optimistic about that part being viable. But that's a minor thing... regardless we need a tree to put other language stuff in there.

petersilva commented 3 years ago

would also be good to include a reference to the organization (the work you have done to come up with this organization) somewhere... under Explanations? dunno... but perhaps just a pointer to this issue is enough.

petersilva commented 3 years ago

so I just noticed... the current directory is called doc/ and your proposal has the root at docs/ so that means the plan is a gradual transition to the new tree, and once it is ready, we just delete the old one? That makes sense to me.

petersilva commented 3 years ago

on v03_wip, doc/fr is completely v2. it has no more relevance to v3 than the v2 documentation does.

MagikEh commented 3 years ago

*Spaces in names*	Sounds good, I'll Camel case everything with the idea of it being subject to change by anyone at any time.
*Debian build*	I'll add that to my list of things to pay attention to.
*Localization subtrees*	I can't think of any good reason why the sub tree cannot be translated as well. The wiki presents everything flat ( `MetPX/sarracenia/wiki/<pageName>_<locCode>.rst`) so it makes good sense to have subtrees translated if they'll be traversed by people code spelunking via terminal or otherwise folders.
*Deepl Translations*	Dang, I was hoping the hype around them being better than gTranslate was a fair margin. I suppose each has their own. Maybe in a few years I'll be able to do it as a part of my French training.
*Organization of project*	This one will likely reside under `/docs/Contribution` as it has to do with helping people work on the project while all other documentation is to help people work with the project. (devs vs users/support)
_Doc vs Docs_	I like it, having a fully seperate folder for the future work will allow the transition to run smoother. I'll branch off v3_wip and make changes only in a new folder.

MagikEh commented 3 years ago

Just realized one flaw with translating file names. Initially it was thought that users would be able to easily move to a localized version of the current file (through the web interface) by just appending _fr to the end of the file name in the url. Since the wiki presents things flat, they would automatically be shown the associated file located in /Localization/fr/

This also means that the sidebar links will also need to be translated. (rather than just foreach appending _fr) Which is less friendly to automation of translation.

So we come up against two use cases.

Do we cater to those working directly in the folder structure (rename files to localized)
Do we cater to ease of use through the web wiki (append _fr to url) and automated translation

petersilva commented 3 years ago

hm... what do wikipedia do? I think they just have completely different trees that have "similar" content...

MagikEh commented 2 years ago

Need to pick, stick to and enforce a doc-strings format.

reStructuredText /is/ our doc extension of choice... but imo it's the most difficult to quickly parse in unformatted text.

petersilva commented 2 years ago

Great idea! I think it should be a separate issue on it's own. that way we can track it from early discussion through decision, and hopefully the addition of automated checks for enforcement.

petersilva commented 2 years ago

Perhaps make "Doc restructuring" a project, and the the directory part is something you have already done? or anyways just moving the directories is one thing that could be marked as "solved" if it is... Getting all the documentation migrated to the new structure, is perhaps a separate issue (in the same project) and it could even be per issue documents... could include translation of docs as separate issues as well.

This is a bit big to cover in one issue.

petersilva commented 2 years ago

or keep this around as overarching, making tick-boxes for sub-issues... and when all the sub-issues are done, so is this. either way... a bit more structure, and more encouragement would come from being able to see progress without having to complete the whole thing.

tomkralidis commented 2 years ago

How about creating docs using a static generator like Sphinx. Sample table of contents for consideration/inspiration:

   overview
   quickstart
   pubsub-concepts
   how-sarracenia-works
   installation
   configuration
   administration
   running/server
   running/client
   extending
   development
   contributing
   support
   license

Overall, the docs should be very low barrier and approachable, with super simple starting points for new users.

petersilva commented 2 years ago

@tomkralidis pointed out some examples also:

tomkralidis commented 2 years ago

https://docs.pycsw.org (doc sources at https://github.com/geopython/pycsw/tree/master/docs)
https://docs.pygeoapi.io (doc sources at https://github.com/geopython/pygeoapi/tree/master/docs)

petersilva commented 2 years ago

OK, @MagikEh and I chatted together... tried turning on github pages, but they only render .md and we are using .rst so likely thing to use is sphinx. Confirmed by looking at Tom's examples, we don't understand how they work, but we understand that sphinx-build is invoked. So first step is to figure out a github action that does the sphinx build and renders the site.

tomkralidis commented 2 years ago

GitHub Pages
- assumes a dedicated branch (i.e. gh-pages) is static HTML, or Markdown, which is automagically generated by Jekyll
- is more appropriate for simple docs or websites
For docs in serious projects, being able to browse docs across multiple versions is valuable, in which case Read The Docs is much more powerful (the pycsw and pygeoapi docs examples are powered by RTD). This is setup on the RTD website and then made available at https://MYPROJECT.readthedocs.org. One can also define a CNAME in RTD to a dedicated URL/subdomain for custom URLs

A scaffold PR is in #460 to create the Sphinx / docs setup. Once this is hardened/fulsome, then integrating on RTD is straightforward.

petersilva commented 2 years ago

ok sphinx structure implemented... now how to import "old" tree so it is there while we build new API stuff.

petersilva commented 2 years ago

OK "old" Actually ( @MagikEh 's new one!) tree is back, but we lost the footer, but we gained search... @MagikEh take a look does this new look cover kind of what you were looking for? we've got search now... but I've got concerns...

in how2guides... roughly half the items in there are about v2 to sr3 upgrading... I feel like that is a small audience and the stuff should be less prominent (irrelevant to most?)
basically all the docs need heavy review... This is starting to be a concern as now that the code is shaping up, this is the next thing to get in shape for release.

petersilva commented 2 years ago

oh and the other cool thing is the local rendering... now you go into docs sub-dir, and do::

make html

and it will get built under build/html ... so you have a local dev cycle without having to publish to find out how the tree hangs together. You need to install python3-sphinx to do that though.

tomkralidis commented 2 years ago

Yes, pip3 install -r requirements-dev.txt will install sphinx.

petersilva commented 2 years ago

ok, so now all the documentation is available under docs/source, including jupyer examples, and it is rendered properly. The mechanism looks great. Now for the hard part, we actually have to review and update the content.

petersilva commented 2 years ago

I spent quite a bit of time restructuring and adjusting things like TofC generation, and also some updates. It's looking a bit better now...

petersilva commented 2 years ago

Last week I was both awestruck and shell-schocked by the new stuff. It was great to finally have a documentation tree that showed everything we have, but also a bit discouraging how many formatting problems there were. I spent a lot of time getting the tables of Content to look OK, and have done some re-structuring. but we need to Review/Proofread the whole thing now.
There is mention of a wiki in docs/source/Contribution/Documentation.rst ... I don't see the point of a wiki on github, where everything is editable anyways, and I always heard people with complaints about wikis being that there was no process for vetting changes... but that's exactly what a PR gives us. So I think the wiki stuff likely should be killed off... but discussion about whether there is a niche for it or not would be good.
Theme? there are a bunch of them... initially used classic, thought it looked kind of old fashioned... now using 'alabaster' ... Is this something worth discussing? anybody I have views? https://sphinx-themes.org/ I don't have any strong opinion... also... read the docs? perhaps that is a good target compatibility there? eventual uplink there? https://readthedocs.org/

petersilva commented 2 years ago

doc/ still contains a lot of old stuff not moved to docs/ Review there also.

petersilva commented 2 years ago

also worry about wall of text effect on the main first page, and the Overview. not friendly enough. need some pictures, logos... etc...

petersilva commented 2 years ago

470 add jupyter notebook about polling.
474 Add config examples.

petersilva commented 2 years ago

@MagikEh 's note about ". error:: Need Peter to help ID worthy information in doc/design to pull over to /docs/Explanation/Design/" is mostly done in ( in docs/source/Contribution/Documentation.rst )

fa1c854de4877e9253a79989f6b99b7eccd8f41f fa1c854de4877e9253a79989f6b99b7eccd8f41f 7d52c0e3bd7d1324269da58e89be08b002611562

discarded v02 design stuff (still available on v2 branches.) created Explanations/History directory, and moved content that sill has some interest there. Peter's tendency to document how things changed getting ripped out of active documents and put into Explanation/History... hopefully less clutterred.

Also there was discussion about how sr3.1.rst is a monster... dragon slain! 3dd88b16cf1820c8445762758a1f704802b8f0a2

sr3.1 split into:

Reference/sr3_options.7.rst ... dictionary-like list of all the directives in a configuration file.
Explanation/CommandLineGuide.rst ... are more narrative description of usage of the CLI.
Explanation/DetectFileHasChanged.rst
Explanation/FileCompletion.rst

sr3.1.rst remains, and it has kind of summary/overview in reference form. Much less monstrous.

The documentation is likely shorter, as the former all-in-one sr3.1.rst monster actually had a lot of repetition, because had been created by merging pages from the different entry points in v2. with the Dictionary approach, there are no repeates.

The CommandLineGuide is still in rough shape, kind of incomplete, but it should be a lot easier for people to understand and contribute to.

Not entirely done. I guess doc/talks should got to docs/source/Explanation/History ?

Also... I broke man page generation in turning off doc/ so Debian packages, when generated no longer include man pages. Everyone seemed surprised they existed whenever I pointed that out, so I think it's no loss. Ideally, we would get them back. in a doc package. Some day #476

petersilva commented 2 years ago

status:

Reference materials are looking complete and accuratish now.
Overview is ugly.
Command Line Guide needs some work, maybe a second version.
- might want to review and merge "1_CLI_introduction.ipynb" and "Setup_a_remote_subscriber.rst"
- needs pictures, circles, and arrows, and a paragraph on the back of each one explaining what each one was.
should remove references to wiki source/Contribution/Documentation.rst as obsolete/inappropriate. (@MagikEh )?

petersilva commented 2 years ago

discussed at dev meeting today... quality considered... good enough for release... though it can always be better. Removed this item as a release blocker.

petersilva commented 2 years ago

@MagikEh do we still need the branches for this issue? I think what we need is already merged, and we are just working on v03_wip.

MagikEh commented 2 years ago

@MagikEh do we still need the branches for this issue? I think what we need is already merged, and we are just working on v03_wip.

Yeah, if there are branches that have the tips merged into the v03_wip I see no issue in deleting them.

MagikEh commented 2 years ago

See about merging the H2 guides with the Tutorials (consider renaming prior to merging folders.)

MagikEh commented 2 years ago

Dev Meeting Notes:

Doubly checkover the dev install notes so the students aren't stuck spinning

MetPX / sarracenia