Added: MkDocs documentation

noirbizarre commented 7 months ago

Checklist

[x] make test-all (UNIX) passes. CI will also test this
[x] unit and/or integration tests are included (if applicable)
[x] documentation is changed or added (if applicable)

Description of change

This PR implements #853:

bootstrap and scaffold a MkDocs documentation using MkDocs Material
split the docs/manual.md into multiple files and format them
add a .markdownlint.yaml and .editorconfing to hint/preconfigure editors supporting it
add .readthedocs.yaml settings file to build and publish on readthedocs.io
add Makefile recipes for documentation
change all documentations references and links to goss.rocks
add a workflow to lint the markdown and add a documentation preview link to pull requests

Documentation site comes with built-in search, pretty deep links and a lot of other features (that may be added or adjusted later on demand).

Choices

As this is a first iteration, many choices have been made. Most of them can be changed now or later.

MkDocs + Material

The documentation engine is MkDocs with the Material for MkDocs plugin/theme.

Documentations pages (except READMEs, cf. Existing READMEs) have been crafted with this in mind:

rendered readability first. The goal is not to be able to read the raw markdown files on GitHub, but to consult the online documentation. So Mkdocs+Material special syntaxes (ex: admonitions) have been prioritized over raw Markdown ones when they bring readability
pretty URLs, easy to share, easy to remind, easy to rebuild. File names have been chosen with resulting URL in mind (one word instead of dashed slug, explicit anchors so explicit title)
consistency: try to follow the same pattern everywhere (like how commands or tests are described), with the same order (title > details > options > example > notes)

ReadTheDocs vs. GitHub Pages

It is possible to publish the documentation on both readthedocs.io or GitHub Pages. I used both solution and for this project I chose readthedocs because:

you don’t have to provide and support a dedicated workflow to publish the documentation
there were no prior GitHub workflows, so I chose to keep those as minimal as possible
RTFD support preview on pull-request while GitHub Actions deployment is not

While ReadTheDocs supports versioning and internationalization, I chose a single version strategy because:

goss is a CLI first and not a library (meaning there shouldn't be huge breaking changes)
URLs are prettier and easier to user/share (cf. Scripts and schemas)
if someday you want to make this a site including a documentation instead of just a documentation, it makes more sense

The idea is to create an account, use the goss slug for this project and use a use the goss.rocks domain as custom domain. I can help if wanted (you can add co-maintainers on any project in the Admin > Maintainers section).

Python tooling

As it is not a Python project but a Go one, Python tooling has been kept minimal. No project manager, only a requirement file in docs/requirements.pip.

Documentation tasks for local dev have been added to the Makefile. You can:

run a live server with autoreload with make docs/serve.
build the documentation (almost) like readthedocs with make docs.

If not virtualenv is detected, one will be created for you in .venv

Existing READMEs and benchmarks

Existing READMEs have been keep. They are included with the Snippet plugin when required. For the main README, Snippets sections have been used to be able to includes parts and spread them in the documentation.

It is most probable that they will evolve later with usage (and feeling), especially for the main README, where some section may move into the documentation and just be linked in the README.

One reason for moving those content in the documentation: you don't have the same level of tooling and formatting for files shared between GitHub Markdown and MkDocs.

The Benchmarks have not moved into the documentation, as it hasn't been touched since 2016. I think that having a documentation site should lead to disabling the Wiki. If benchmarks are up-to-date and should be published, I can move them into the documentation site.

Examples

Examples have been left untouched, mostly because I wasn't able to find where they are documented or used. I see those:

the root examples directory
the docs/goss.yaml, docs/vars.yaml, docs/myapp_gossfile.yaml and docs/rendered_goss.yaml

They could be grouped into the docs/examples or the root examples directories and used/show-cased into the documentation.

Changelog

The changelog page has been created and just link to the GitHub releases.

GitHub Actions workflow

I took the liberty to add a basic workflow for PR touching the documentation, it will just:

lint the markdown
add a preview link to the pull request

No need to have a "build documentation" workflow as readthedocs should take care of if for you when importing the project and then allows you to fine tune it in a few clicks

Branding

For the moment, colors are very neutral, for both light and dark mode (black header, white or black background depending on the mode). The logo and favicon are the default ones. CSS is minimalist (just used for the platforms support matrix rendering).

It is however possible to customize everything, and in particular the home page if you want to make a landing page like the Material for MkDocs one.

Scripts and schemas

Schema file has been renamed to have a clean URL (should be available on https://goss.rocks/schema.yaml).

In practice, you can expose any static file on the site, including the installation script.

Containers

While dgoss, dcgoss and kgoss have a special status, they are part of the project but not officially distributed with it, I chose to include their READMEs as documentation is a "Containers" subsection has I spent a lot of time searching for them and going back and forth between the manual.md and dgoss README. I believe this will also be helpful for lots of people.

FAQ

I did not yet include a FAQ section has I didn’t have any initial content. Anyway, adding this section later if required should be easy now.

Editors hinting/settings

The .editorconfig will allows editors supporting it to have the proper indentation preconfigured. The .markdownlint.yaml will do the same for the formatting and lint rules (most editors will automatically read it and provide either autoformatting, built-in linting/highlighting and preconfigured snippets).

Note that markdown files have been formatted to comply with those.

Typos

I tried to fix some of them but given I redacted some content, I apologize d by advance for those I introduced. Maybe some day it should be worth to add pre-commit with CodeSpell to either automatically fix them or prevent their introduction.

Preview / try

You can preview the documentation site locally (because readthedocs and the preview on pull-request is not yet enabled) by the checking out this branch and run make docs/serve. However, I can provide screenshots or a zipped site if needed.

In the meantime, just a screenshot to tease:

screenshot-localhost_8000-2023 11 12-19_17_00

Now I am waiting for your feedback and questions @aelsabbahy and @petemounce. Sure it won't be perfect from the start (documentation is a long run) but I think this should be a huge step forward. WDYT ?

aelsabbahy commented 7 months ago

Whoa.. there's a lot here. At a quick glance this is awesome, thank you for taking the time to do this!

I'll have to spend some time with this PR to understand it better.

In the meantime, I do have a small question. noticed the docs/links are https://goss.rocks/... any suggestions on best way to host this. I sit github pages or something else?

noirbizarre commented 7 months ago

To me, you have 2 choices:

github pages: everything is local to the repository, just the publication workflow from this PR. Downside, no preview on PR (yet). Build is added to the org CI time (but shouldn't be an issue given it's fast). Custom domain documentation is here
readthedocs.io: dedicated service, provide build, some tooling around the doc like automatic versioning if you want, preview on pull-requests, but this is an external service requiring an account. Custom domain documentation is here.

Support for both is included in this PR, once you have chosen, I'll remove the part not needed.

I used both, and I would say that in the current case, the only diff would be that readthedocs allows previewing PRs and then it's only personal taste.

Note: for github pages, I talk about the new "deploy with a workflow" way, not the legacy "gh-pages branch" way

I'll have to spend some time with this PR to understand it better.

Run make docs/serve, you'll have a live reload server allowing to test things and see the result in live. It will be way more explicit. Th evolume of this PR is mostly due to MD formatting and docs/manual.md being cut in multiple files

petemounce commented 7 months ago

(I used GH pages via workflow and I never got preview working. That was a major downside in that context since it raised the bar for docs contributions)

noirbizarre commented 7 months ago

PR updated with review suggestions and fixes.

For some, I added an explanation on why it is done like that (either directly in the source code, either as answer in the comment, either both).

It still has both Github Pages and ReadTheDocs support workflows until a decision is taken.

Note that if preview is not yet available in Github Pages, it is planned to and already merged in their codebase (cf. https://github.com/orgs/community/discussions/7730 and https://github.com/actions/deploy-pages/pull/61).

Note: I took the liberty to mark suggestions as resolved when there is nothing more to discuss, in order to be able to focus on the remaining discussions.

Suggestion: Enable Github actions for contribs or it will be hard to review the docs workflow for this PR and harder for contributors to detect issues on their contribution. I saw in #786 that you started migrating to Github Actions but fear to enable it (while you already have TravisCI running for every PR). I would suggest to enable it and enable maintainers to approve first time contributors workflow run. You can also ask maintainers to approve each workflow run if you want. Just know that (I say that as someone which is contributing to lots of projects daily) not having the CI run will make you lose contributors:

makes review/merge cycle way longer
contributors can't fix their CI issue until you run it. If it's not done quickly, contributor will just forget its contribution or have an extra rebase work to make it green or just lose motivation)

(it is often than I submit a fix, the maintainer approve the CI run weeks later and I already moved to something else, changed project or just don't remember the change and so don't have the motivation to get back to it)

Note: I have quite an expertise on Github Actions (this is literally my current daily job and the expertise I sell to companies) and I can help to migrate the Travis CI to it in another PR if you want.

petemounce commented 7 months ago

I appreciate the time you have invested and your explanations; thank you!

re: GHA - yes, I'd like to migrate from Travis. I'm pretty familiar with GHA as well; productivity focused day-job. I lament buildkite does not offer build machines for OSS.

Totally agree on "CI needs to feed back to contributors" points. I believe GH won't execute a new workflow until the file arrives in main. How I've done that before is to separate the CI addition from the code-to-integrate addition. I could pull the workflow in via a different branch and you could rebase; what're your thoughts? (Obviously the workflow will fail on main until this PR lands)

aelsabbahy commented 7 months ago

+1, many thanks for the time spent on this, this is awesome!

Gh pages vs readthedocs.. I don't have a strong preference here as I don't have much experience with either. I'll defer to @petemounce and @ripienaar if they have a preference.

Travis vs GHA, I'm definitely in favor of migrating to GHA. I have to email Travis every now and then to get more free OSS credits.. which isn't a fun experience. I worry one day they may stop supporting OSS completely.

There's a bit of nuance to the way the Goss tests run (systemd in docker, osx, etc.) If that can all work on GHA great.. if not.. then the heavily black box approach using e2e tests may have to be re-evaluated.

aelsabbahy commented 7 months ago

Please notify me when this PR is ready to go.. or waiting on me. This way I can start looking at all the account, hosting, repo settings needed and make the changes.

Again, many thanks for taking on this large effort.

aelsabbahy commented 7 months ago

Hello, hope all is well. Just checking back in to make sure I'm not the blocker. Is this PR waiting on me for anything?

noirbizarre commented 5 months ago

Hi 👋🏼 Sorry for the lack of response lately (overbooked, holidays, children... life). I'll update the PR (make it readhtedocs the backend) by the end of week !

noirbizarre commented 5 months ago

PR updated:

rebased on latest main
removed the GitHub Pages publication
use mkdocs-awesome-pages-plugin instead of the builtin nav
all dependencies are pinned

So it should be ready and will remain:

Connect to readthedocs using your github account, import the goss github repository and add goss.rocks as a custom domain (with Canonical option disabled)
Enabled github actions on this repository

And that's all I think, you should have a new website after that 🎉

aelsabbahy commented 5 months ago

Awesome, thanks for the update.. I'll try to get around to this as soon as possible. I just had a newborn so life is a bit hectic, but this PR will be my priority when my time frees up.

This is amazing work, thank you so much for taking the time to do this.

petemounce commented 5 months ago

I just had a newborn so life is a bit hectic,

Congratulations!!

noirbizarre commented 4 months ago

I just had a newborn so life is a bit hectic

Late congratulations !!! Enjoy, your newborn is more important than this PR !

PR has been rebased and conflict solved, just in case you find time to merge. You can activate the readthedocs part later if you want to merge ASAP, so people can't contribute to the split documentation to avoid conflicts.

aelsabbahy commented 4 months ago

I just had a newborn so life is a bit hectic

Late congratulations !!! Enjoy, your newborn is more important than this PR !

PR has been rebased and conflict solved, just in case you find time to merge. You can activate the readthedocs part later if you want to merge ASAP, so people can't contribute to the split documentation to avoid conflicts.

Yeah, that sounds good. Going to merge this. I'll probably reach out to you in the coming weeks to finalize the readthedocs part of this if I have any questions.

Thank you for the hard work on this, this is awesome and has been requested for a while.

aelsabbahy commented 4 months ago

Do you have the previous commit that uses GitHub pages by any chance?

If the commit exists somewhere in your repo can you drop a link on here if it's not too much trouble.

Mostly asking for my own education/growth.

aelsabbahy commented 4 months ago

After merging, I see the following CI error:

https://github.com/goss-org/goss/actions/runs/8087101681

goss-org / goss