Find a solution for versioning documentation via ghpages

[ruebot]

Currently, the way we use MkDocs and ghpages, we don't have a way to version documentation. As we are moving away from the 7.x-2.x branch, I would like to see that documentation still exist, but, at the same time, we need documentation for the master branch.

Let's use this issue for strategizing a way accomplish this.

One option to kick off the discussion would be to just move the 7.x-2.x documentation over to the master branch in a directory, and map it out from there.

[ruebot]

https://github.com/mkdocs/mkdocs/issues/193

[kimpham54]

can we use git tags to version docs in master? then in master have a directory for each release

[ruebot]

@kimpham54 maybe. But, the repo includes all of the 7.x tags as well, and we don't have any tags for 7.x-2.x. The @Islandora-CLAW/committers will need to determine if we should "release"/tag any of that work, of if we just move on from it and abandon it. We'll probably what to draft a policy for how we handle documentation moving forward. If that is something you want to work on, and @dannylamb is cool with that, I think that would be a good start. All that said, the August sprint is probably going to be hyper-concentrated on the MVP.

[kimpham54]

OK, I can take this on this sprint but plenty of work to do still with #281 so happy to wait as well

[dannylamb]

@ruebot at least tag it. we're not just going to nuke it.

@kimpham54 handling documentation per version is a very valuable goal and could be done independent of mvp design. i'll put an item on the agenda for the claw call if you have any ideas you want to share, or if you want to talk strategy with anyone.

[ruebot]

@dannylamb might as well add an agenda items about what we're going to tag then, because we'll definitely need to talk that out.

[dannylamb]

@ruebot done

[ruebot]

Another option to consider: http://couscous.io/

[kimpham54]

will look into https://help.github.com/articles/configuring-a-publishing-source-for-github-pages/

[ruebot]

@kimpham54 we're already setup for that with MkDocs; http://islandora-claw.github.io/CLAW/ is ghpages.

[kimpham54]

I looked into the possibility of viewing deployed tagged versions of documentation by tagging master in gh-pages. This doesn't seem possible from what I can tell. As an alternative proposal to view previous versions, we can create a new directory to organize the documentation for each new release:

7.x-2.2 (current) accessed at http://islandora-claw.github.io/CLAW/7.x-2.2/ --| What is Islandora? accessed at http://islandora-claw.github.io/CLAW/7.x-2.2/about --| What's new? --| Contributing --| Chullo --| Alpaca --| etc. 7.x-2.1 accessed at http://islandora-claw.github.io/CLAW/7.x-2.1/ --| What is Islandora? accessed at http://islandora-claw.github.io/CLAW/7.x-2.1/about --| What's new? --| Contributing --| Chullo --| Alpaca --| etc. 7.x-2.0 accessed at http://islandora-claw.github.io/CLAW/7.x-2.0/ --| What is Islandora? accessed at http://islandora-claw.github.io/CLAW/7.x-2.0/about --| What's new? --| Contributing --| Chullo --| Alpaca --| etc.

Tagging can still be useful if a tag is created for a release, it can trigger a webhook to deploy the new version of the docs to gh-pages. You can also use webhooks to deploy docs on a regular basis between releases.

Webhooks can also easily let others contribute to the documentation - once a doc contributor proposes a file change via pull request which once this is approved it can be deployed to gh-pages.

Proposed Workflow:

1. In preparation for a new release, create a directory for the new release docs and copy all existing docs of the latest release into the new release directory
2. Work on the docs for the new release
3. When a new release is tagged, run script to deploy docs to gh-pages (webhook)
4. When documentation is updated or a pull request is made to update /docs/*, run script to deploy docs to gh-pages (webhook)

[kimpham54]

@dannylamb if you have any comments or suggestions, i'd be happy to look into other options for this sprint

[whikloj]

Would it be helpful if we set up a test repository, and actually try some of these options out? Like with a fake release?

[kimpham54]

sure, I could work on that

[kimpham54]

I created a copy of the islandora docs in my own repo, https://kimpham54.github.io/CLAW/. I think I understand the crux of the workflow now.

I see now that one of the issues is that the docs have symlinks to READMEs in other repositories, so to version those docs could be a pain. Is it an option to have all of the docs just in the CLAW repo, without symlinks? Then the problem becomes how to keep each component's README in sync with the READMEs in CLAW/docs.

If someone's available tomorrow I'd like to understand how the docs are being deployed to gh-pages right now, because I'm not sure the way I'm doing it is the way that that the docs are actually being updated.

[ruebot]

@kimpham54

I'd like to understand how the docs are being deployed to gh-pages right now

https://github.com/Islandora-CLAW/CLAW/blob/master/docs/technical-documentation/docs-build.md

[kimpham54]

@ruebot thanks nick. just a couple of questions:

1. when i deployed mkdocs in CLAW, it gave me a WARNING error: "The page "alpaca/README.md" contained a hyperlink to "alpaca/CONTRIBUTING.md" which is not listed in the "pages" configuration." to fix this, i cloned all of the symlink repos (alpaca, chullo, crayfish, etc) and copied them into CLAW so the symlinks in the docs/ points to the READMES of these repos. I'm guessing this would be pain to do to everytime you want to update docs. Is there another way to do this that I haven't figured out?
2. When the site/ builds, do you push this folder into the gh-pages branch? Or should this happen automatically? gh-pages doesn't seem to update when i run mkdocs gh-deploy --clean, and i see that site/ is in .gitignore

[ruebot]

@kimpham54

1. The symlinks are in the repo itself with the submodules. Did you update the submodules when you did your clone of CLAW git submodule update --init --recursive?
2. Happens automatically. .gitignore isn't in the gh-pages branch: https://github.com/Islandora-CLAW/CLAW/tree/gh-pages

[kimpham54]

@ruebot

1. ah! thank you
2. i realize now that i had to change my remote name origin to point to my own fork for it to update gh-pages.

going to look into this more tomorrow, thanks

[kimpham54]

A first kick at the can: https://kimpham54.github.io/CLAW/

This is how the docs are structured: https://github.com/kimpham54/CLAW/tree/master/docs (symlinks aren't fixed in 8x12-currentRelease, I couldn't figure out how to do it yet and so the docs didn't all build)

I think symbolic links should be removed, and instead use webhooks (? if possible) to mirror the READMEs in each submodule repository to the READMEs in CLAW/docs/[submodule]/README.md. That way all documentation can live in one place, which will be simpler for contributors should they wish to add additional .md files. They would add it to docs/[submodule] instead of the submodule repo itself. You wouldn't have to keep making symlink docs in CLAW/docs. Submodules would only have READMEs that stay in sync with the READMEs in docs/.

Then, for every release:

1. deprecate the current release folder (remove the -currentRelease suffix)
2. copy that folder, give it the new release version name, append the -currentRelease suffix, carry on with life

Notes:

• mkdocs didn't like naming of the directories 8.x-1.0/ 8.x-1.1/ it collapsed those top level folders. had to change it to 8x10/ 8x11/
• removed pages to mkdocs.yml - not ideal to have to manually add pages to discover
• certain themes don't seem to work well with this directory structure. would likely want to add extra css/js files to customize the theme you decide on

Any comments or other suggestions on what I should look into next are welcome

[ruebot]

Looking at the navigation here, I find it very confusing.

mkdocs didn't like naming of the directories 8.x-1.0/ 8.x-1.1/ it collapsed those top level folders. had to change it to 8x10/ 8x11/

This would have to follow semantic versioning, given the Versioning Policy

---

...what if we took a step back, and looked at this is a different way. Is there anything that says that we have to have a full set of documentation for every version? Would it be too naive to say that we just have documentation, and if we have any version specific documentation, we wrap it in version headers or warnings. Then, go through a deprecation cycle, just like we would with code?

[kimpham54]

if you want documentation by release, we could probably use custom css to override the way that the links are being displayed in navigation. we could collapse the subheadings and links.

the solution is a little complex, so maybe it's not the best approach to managing documentation if having full documentation for every release is not required. i guess it depends on how people intend to use the documentation.

instead, you could have release notes highlighting what changes have been made, what documentation has been changed and deprecating

[whikloj]

Hmm... I think we need to work on the sidebar for sure. Best to collapse entirely the non-selected version.

Also, when I choose the Home link under a release it only seems to highlight part of that section. It doesn't include the sub-repos.

Lastly, we might need to re-write links so the Alpaca link under 8x-11/Home points to the 8x-11/Alpaca/README.md, currently it points to github.com (which is the current version).

[rdeanlib]

Have you considered Read the Docs? It seems like it can do what you want. http://docs.readthedocs.io/en/latest/versions.html

Example from Omeka: http://omeka.readthedocs.io/

[kimpham54]

@rdeanlib I'll take a look at this - thanks!

[kimpham54]

Demo of CLAW documentation by tag using readthedocs: http://islandora-claw-kimpham54.readthedocs.io/en/latest/ (you can switch to different versions of the docs by clicking on "Read the docs" link at the bottom left hand corner.

For now it seems like readthedocs isn't pulling in the READMEs from the submodules like mkdocs alone: https://kimpham54.github.io/CLAW/ - this might not be an issue if you want all your docs to reside in one repository.

Link to the readthedocs project: https://readthedocs.org/projects/islandora-claw-kimpham54/

Does this seem like a better approach? Again comments and suggestions are welcome

other issues i've noticed:

• edit on github on the bottom left corner doesn't link to the proper file on github: https://github.com/mkdocs/mkdocs/issues/269
• readthedocs overrides what theme you specific in mkdocs.yml: http://stackoverflow.com/questions/37952916/can-i-use-external-themes-with-mkdocs-on-readthedocs-org

sphinx seems to be better supported with readthedocs than mkdocs, so that might be worth testing?

[kimpham54]

So I've been playing around with some of the different options again, and would like to see if any of these options stand out for anyone in particular to pursue. I've included some of my thoughts on each platform below.

readthedocs: restructuredtext (sphinx)

• benefits: readthedocs is most compatible with sphinx. allows for versioning, includes many additional modules and config options and semantic markup
• drawbacks: more complex system than mkdocs, new syntax to learn and will need to use .rst files.

readthedocs: mkdocs

• benefits: allows for versioning
• drawbacks: many bugs to still work out, direct edit link doesn't work out of the box but there are examples of where it works

mkdocs standalone

• benefits: simpler architecture, can easily edit, docs can be managed in separate repos if desired
• drawbacks: no native support for versioned docs

FEATURE REVIEW

easily view and edit different versions for release documentation mkdocs - no, unless you do so from github sphinx (readthedocs) - yes mkdocs (readthedocs) - yes

easily edit and clear path to do so from html page to github repository mkdocs - yes sphinx (readthedocs) - somewhat mkdocs (readthedocs) - no

notes:

• you can't edit tagged documentation e.g. https://github.com/kimpham54/docstest/blob/1.0.0/docs/crayfish/README.md, but you can edit documentation if they were on versioned branches

flexible enough to support docs all in one place or docs embedded with code in separate repos mkdocs - yes sphinx (readthedocs) - yes mkdocs (readthedocs) - yes, but needs to be tested

notes:

• docs in separate repos will need to be linked
• docs in separate repos will be harder to manage if you want people to contribute - cloning multiple repos

[kimpham54]

also is there any interest using confluence again?

[ruebot]

:-1: Confluence

[whikloj]

Awesome job @kimpham54, regarding the question of "easily edit and clear path..." you said "somewhat" for sphinx. Could you elaborate a little?

[kimpham54]

@whikloj I certainly can. Sphinx's use of reStructuredText at first glance has similar, easy-to-read syntax as markdown, but if we want it to be it can become a lot more complex using semantic markup (http://www.sphinx-doc.org/en/stable/markup/index.html) that isn't supported in Github. To get started with Sphinx, there are also a lot of additional built-in modules and configuration setup pages. It doesn't mean we have to use the complex syntax but it's there as a possibility and if someone decides to use it, it may be harder to maintain or convince other contributors to follow those conventions. It's also worth noting that when I did a quick search for package support in sublimetext, atom, and repos in github for markdown vs restructuredtext it also seems like markdown as a general format is more widely used.

[kimpham54]

I was reviewing this ticket and wanted to provide an update on versioning with github pages. @ruebot referenced this ticket https://github.com/mkdocs/mkdocs/issues/193 earlier in the conversation, which is not yet resolved. The versioning feature based on github tags works with readthedocs, however readthedocs only supports sphinx and its own flavour of mkdocs (not compatible with newer versions of mkdocs nor with the material theme that we are currently using).

I'm not sure how to move this ticket forward, other than to suggest that if we have those processes in place (such as coordinating tags across repositories for releases) that would be a step in the right direction. Does anyone have experience with automating this via github webhooks or something similar?

[ruebot]

Might be a good topic of discussion for the documentation team at the hackfest here in a couple weeks. @manez, that sound ok?

[manez]

@ruebot we can always suggest it for a topic 👍

[cirosantilli]

I have seen the light, use single page README.adoc: http://www.cirosantilli.com/markdown-style-guide/#use-asciidoc

[f-w]

I faced same challenge and solved it using GitHub actions. Details is described in this stackoverflow answer.

[dannylamb]

Apparantly the material theme we're using has this as a paid feature: https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/#versioning

[manez]

We are now subscribing to the paid features, but have not implemented this one.

[eldonquijote]

@eldonquijote tested MkDocs Material versioning with mike and found it to work. Requires changing the Travis configuration to invoke mike instead of mkdocs.