External documentation must respect versions from coredev/ release

[jensens]

This is a copy of issue plone/documentation#275 moved here because its a papyrus related issue

Story

I have this PR #192 open and in there we figured out to better include buildout.coredev documentation instead of repeating ourselves here.

So I tried to add the buildout.coredev/docs to externals.txt like this:

https://github.com/plone/buildout.coredev.git,source/documentation/develop/coredev

Problem

This fetches the 4.3 docs (because its set as default branch). I don't see a way to set the branch to 5.0.

This is valid for all external packages.

Criticism

• We have no version pinning for documentation.
• I have no idea why we invent a new way to fetch things from git. Its like reinventing the wheel.

Ideas/ solution

Why don't we use mr.developer and the sources files in buildout.coredev? For some (what we consider core docs?) we already do.

• read sources file from buildout.coredev in the version needed (buildout extends)
• provide checkouts file for the to-be-included sources.
• or take eggs from releases and fetch docs from there (i'd prefer this much to have docs matching the release and additional '4.3-latest' + '5.0.-latest')
• have some mini-recipe to symlink the stuff (or some script)
• symlink docs folders to the locations in source tree?
• no git fights with sparse (worked for some reason not on my maschine/ git version)

@svx answered:

Hi,

Ok let me explain:

We do have versions of the docs for different versions of plone, all of this is done in https://github.com/plone/papyrus here we have branches for version 3,4,5 and in this branches is also configured which version of the docs should be pulled, this is maybe not ideal but it is proven to work.

Release of docs have also a git tag, at least that is on our list, but there tagged with the release date of the docs, and the reason for that is,with that tag, the build server builds automatically pdfs, zeal/dash/, docker cts and lots of other stuff.

Git-sparse is in git since version 1.7 in the ubuntu world that means since 12.04, I am quite sure that you are developing on a newer version :)

This said, I completely agree spars is not perfect, and it is sometimes hard to use, but it is also not something what is rare to use, it works.

Some other points you mentioned are really interesting and I certainly would like to test them, but please keep in mind, the docs 'core' team is quite small we are two people at the moment, so we simply have a time issue. Second, we also have to keep in mind, that not all of the user are developers, so we also have to try to think as a user, this is hard, but the docs in general are not developer only, this may not be the case for core devs, but imho we have to find a sane way to work on both equal easy.

At the end I also want to say, that the whole docs project is also, like the readme of papyrus says, work in progress and we are learning and figure things out all the time, and yes sometimes we do not get it right straight form the beginning, but at least when I look what we currently have and how other projects doing it [as we talk and work together with other docs writers, even with professional ones] we maybe not perfect but we also not doing that bad.

[jensens]

Goal in short:

• http://docs.plone.org online plone-documentation must follow always the latest.
• additional following the release would be great, but not mandatory

I looked a bit further into the issue.

Current:

• papyrus has branches for 3/4/5 for the core docs
• external added docs from https://github.com/plone/papyrus/blob/master/externals.txt are always fetched from the current default branch and so probably the wrong version.

Implementation for latest :

• get rid of unflexible externals.txt and its make section
• use sources.cfg from buildout.coredev (3/4/5)for the latest
• add a checkouts-addons.cfg adressing the packages we need in papyrus
• use mr.developer 1.33 to only clone tip of branch/tag with depth=1 (speedup, less disk space wasted)
• symlink (not windows compat, do we need to be?) or copy into the docs source tree. This can be a mini-recipe or just using cns.recipe.symlink or plone.recipe.command.

Implementation sketch for release fixed docs (much more effort):

• use tags of packages to build docs for releases. versions can be taken from the releases versions files. version number in version is always the same as the tag as it seems from a quick check. (thx zest.releaser!)
• figure out a way to make clones from [versions] section.
• OR figure out if theres an easy way to auto-create a version-[4.3.6|5.0|...] file in buildout coredev on the tag while releasing and use this (so implementation for latest works for releases as well - i'd prefer this, even if it add an additional step to the release process)

Responsibility

• there are no resources implementing this in the docs team.
• I can implement this for latest - this is very straightforward - if the idea is accepted.

[svx]

Hm,

I guess I do not understand what you mean,

• docs.plone.org shows always the documentation for the latest version of plone.
• papyrus has different branches and in this branches is also defined which version of the documentation repo is fetched, so far this works and if you are in the branch of papyrus for plone 5 you wil fetch the docs branch for plone 5. Can you please explain where this is wrong ? Did we forgot to check this in one branch ?

I do not understand why you only focus on buildout.coredev. the docs are more than coredev, can you please explain ?

I do not understand what you want to do with tags and build releases can you please explain ?

And I certainly do not see our current setup as a bug, there is place for improvements sure ! But labe this issue as bug, I certainly do not agree :).

[jensens]

look, i.e. plone.app.theming is always at master.

[jensens]

btw, i have a simple solution to use the externals from the matching coredev with mr.developer and a simple generated script. this would simplify the whole a lot too.

[polyester]

I think there's a misunderstanding at play here. But let's see your solution, @jensens and see what it does.

I have a feeling it may have to be tweaked, since there is a specific reasoning behind which packages go where in the docs, and which ones we currently not include at all. But it's much easier to talk if you make a papyrus feature branch, so we can all see what it actually does.

Because now we're at semantics and misunderstandings, which does not clear the picture.

[jensens]

yes, more at semantics. i'll prepare a solution on a branch. stay tuned.

just another try to explain: plone.app.theming uses 1.1.x branch for 4.3.x - ans so the docs there targeting 4.3.x, right? https://github.com/plone/buildout.coredev/blob/4.3/sources.cfg#L112

but it targets master for 5.0 https://github.com/plone/buildout.coredev/blob/5.0/sources.cfg#L76

papyrus external.txt does not take branches into account. so the plone.app.theming addon docs are the same master docs for both docs.plone.org (4.3 and 5.0).

[polyester]

Yes, that makes it clearer. It was always meant for externals.txt to have versions (it has some different lines for instance for 4 and 5), we just never fixed the versions proper. So yes, they should be pinned.

Only, we fetch more packages than just mentioned in coredev, we also fetch for some add-ons. And, we really don't want full packages, just the /docs dir (and readme.rst + changes.rst), since otherwise it messes up the translating setup and makes it extremely confusing for non-developers to contribute to the docs.

[svx]

Ah now I get it !

yes that is right, currently papyrus does not take care about that, yet.

I would like to see that fixed, but there is one important step to keep in mind, and this is partly the reason why we not dealt with that so far.

We have to make sure that all the docs of the different version are ok, meaning we have proper docs for all versions of the includes, following our best practices and style guidelines. How we make want to make sure we have this in a short period of time?

If we do not do that, our quality we decrease a lot.

I am looking forward to your branch !

[jensens]

Here we go https://github.com/plone/papyrus/tree/jensens-external-refactoring This is the rough idea including sparsing the git repos (which tooks the most time btw.) Also make html works as it seems, but i did not check translations and all the other shiny but complex stuff.

[jensens]

some remarks:

• also sparse git fetches always the full history as i can read from the confusing documentation
• sparse just shows a modified view on the repository
• --depth seems to have its problems with branches, at least it failed when i tried (even after updating to latest stable git release).

[svx]

@jensens please remember when we decided to try out and use git-sparse [Munich sprint year] there was not the shiny new version of mr.developer, so we had to go with something what was already there.

We knew from the beginning that is was not really perfect had had/has some issues, for our use case.

So yes, it is not perfect but well for the beginning it was better than other possible solutions. Please also see that at this point last year, it was not quite clear where we will go with the docs, that is something what evolved during the last year, through learning, getting knowledge, experience and talk with other documentarian.

[svx]

btw: maybe you already saw it:

https://github.com/plone/papyrus/issues/54

As you can see we already started to lock into the new version of mr.developer .....

just saying :)

[jensens]

I know. The problem here is not mr.developer, it is that git clone --depth is quite complex and its behavior is not all the time what you would expect at the first place.

[jensens]

@svx Also dont take it too much as critics, the idea to collect all the docs from the different sources is great and improves the docs a lot!

I just want to offer a solution for the problem described above using our usal tool chain of buildout and mr.developer plus some additional declarative generated scripts and so reducing the overhead of/ time used by the documentation team by pointing to sources/branches already maintained by the developers. The only major difference is the symlinking used in that approach

What do you think about the solution? Shall I turn it inot a Pull Request?

[svx]

It definitely looks really nice and interesting that for sure, I had no time to play with it yet, I will do that next week, maybe we could use to start your solution on Plone 5 documentation, I will have a look after Tokyo, meaning in the next 6 days.

[jensens]

take your time.

Sure, we can start using it for Plone 5.

But with this approach we even dont need to have papyrus branches for different versions of the docs anymore, we can just need for each docs-version a different checkouts-documentation.cfg, like checkouts-documentation-3.3.cfg, checkouts-documentation-4.3.cfg, checkouts-documentation-5.0.cfg and so on. In its details this need some hirnschmalz (deeper thinking about it). But it would reduce complexitiy in terms of multiple parallel branches here.

[svx]

@jensens can you write some docs how 'assemble-docs.tpl.genshi' works ?

[svx]

@jensens Would you mind to document or explain this part too:

[assembly-information]

first is order for symlinking,

divided by dash,

second is either

the package path to documentation in mr.developers sources-dir

or a COMMAND (executed before the sym-linking)

1-COMMAND = mkdir -p ${buildout:docs-dir} 1-documentation = documentation 2-bobtemplates.plone/docs = documentation/develop/addons/bobtemplates.plone 2-buildout.coredev/docs = documentation/develop/coredev 3-COMMAND = mkdir -p ${buildout:docs-dir}/documentation/external 3-ansible-playbook/docs = documentation/external/ansible-playbook 3-collective.transmogrifier/docs = documentation/external/collective.transmogrifier 3-diazo/docs = documentation/external/diazo 3-plone.api/docs = documentation/external/plone.api 3-plone.app.caching/docs = documentation/external/plone.app.caching 3-plone.app.contentlisting/docs = documentation/external/plone.app.contentlisting 3-plone.app.contenttypes/docs = documentation/external/plone.app.contenttypes 3-plone.app.dexterity/docs = documentation/external/plone.app.dexterity 3-plone.app.multilingual/docs = documentation/external/plone.app.multilingual 3-plone.app.robotframework/docs = documentation/external/plone.app.robotframework 3-plone.app.testing/docs = documentation/external/plone.app.testing 3-plone.app.theming/docs = documentation/external/plone.app.theming 3-Products.TinyMCE/docs = documentation/external/Products.TinyMCE 3-tutorial.todoapp/docs = documentation/external/tutorial.todoapp

this numbering thing is just confusing me at the moment and if I try to add or change something it will break... we have always to remember, the idea is, that also less technical people are able to contribute to the docs, meaning we need to document what, why, where and how we are doing stuff and/or use solutions which will also work for people like me for example :)

Thanks for your implementation I am still playing with it ! :)

[polyester]

After a hand-merge and some extra documentation, this is now merged with https://github.com/plone/papyrus/commit/85431194f301caa9f6154588461ebd375e114f6f