Discussion: separate out robot-screenshots

[polyester]

The generating of robot-screenshots is nice, but it is also very fragile, and is hindering upgrades. See also https://github.com/plone/papyrus/issues/118#issuecomment-263045817

So, the question is if it would be a better idea to separate them out into a separate test-suite; that would only depend on Plone, not Sphinx. The resulting screenshots could then be uploaded into a repository, and the screenshots pulled in as just that, static images, for the docs. As far as Sphinx is concerned, they are just a bunch of images in a folder.

Advantages:

• separate docs-building (Sphinx, tests with coala, containers), this greatly reduces complexity for the actual doc-building, as a lot less eggs need to be fetched and less software needs to be synchronized and pinned.
• for the screenshots, 'all' you need is a working combination of version pins for robot-testing, skipping the Sphinx integration
• separation of concerns for doc-authors: if you include a new piece of documentation, you can start with a 'hand-created' screenshot. Then people with more test-writing skills can make that into a robotscreenshot at a later time.
• the writing of the screenshot-tests does not have to be 'artificially' put in Sphinx-like mode; it more closely aligns to how other robot-tests are written, making it more recognizable to experienced test-writers.
• (maybe?) the tests/screenshots could even live in CMFPlone itself; at least then the whole version-number stack of robot/selenium would remain consistent with the other robot-tests that exist. Even when that's not possible and a separate repo/buildout is needed, it should be easier to re-use the same version pins.

Disadvantages, or points to take care of:

• we'd need a clear naming scheme for the screenshots. Currently they have 'mnemonic' names, but a numerical scheme is also possible. We could even include Plone's version in it, so 5.1-0001.png. Although we only get one shot at thinking up a good naming scheme, so that needs to be thought out well; maybe to be future-proof we should include language as well, so ploneversion-lang-number.png. (Just a suggestion, needs thought)
• it's a one-time time-investment to rename/renumber the shots, and take out the robottests currently in the docs; is that worth backporting to Plone4 and/or 5.0, or just do it for 5.1?

As said: this is a proposal up for discussion.

I'm looking for a way to keep the advantages of robot-screenshots, while getting around the significant disadvantages we have seen.

[svx]

I wrote more or less the same.

Also, I have a almost flawless working papyrus 1.5 alpha 0.0.1.

Besides a good solution for robot-screen-shots all what is over to fix is a better coala setup, which should be painless easy once there is a new coala release, since I do not want to use pre releases for production.

The other thing to test more, is a new script which checks you commit and runs checks against included .rst files, if there any. I am testing and tweaking that currently in production at my day job, there are still some hick-ups and bumps but it is getting better.

Building and deploying docs is done via CI, meaning if tests are passing a new docker container will be build, tagged, pushed to a registry and automatically deployed to the host server. Here we use a 'green-blue' proxy to move all open connections over to the new container,stop and remove the old one. Since the old one is still in our registry we alway can do a rollback if we need. A point to think of is proper tagging.

This is maybe the overkill and we do not need to transfer the connections, but well, I like playing with it. I use that setup since a couple of month in production and its working well so far.

[MrTango]

I like the idea of separating the screenshot generation. Also because of problem combining the robotframework code with the spinx code see: #129.

[svx]

We did it for the docs ! Done here, but this still needs to be setup on Jenkins @polyester @gforcada can you have a look when you have time ?

[gforcada]

Yes I can have a look, where are the robot tests now? In Cmfplone? Is there a special way to trigger them?

[polyester]

they live at https://github.com/plone/Products.CMFPlone/tree/master/Products/CMFPlone/tests/robot/robodoc, the easiest way to trigger them is from a full buildout.coredev instance where you run bin/pybot -v BROWSER:phantomjs src/Products.CMFPlone/Products/CMFPlone/tests/robot/robodoc

(when using phantomjs as the browser). I'll be experimenting with headless Chrome as a driver, but phantomjs works well.

[polyester]

The idea is that we don't run them at every jenkins run, because they are expensive/long to run, but let's say once a week (in the future), and then the results get pushed into https://github.com/plone/documentation-roboshots