sphinxcontrib-robotframework does not generate 'hard' errors

[polyester]

hi @datakurre , would it be possible to have sphinxcontrib-robotframework generate hard errors (meaning sphinx exits with error code) if the screenshot tests fail?

tried sphinxcontrib_robotframework_quiet = False, but sphinx will still churn along when there are failures in the tests. Some setting like "sphinxcontrib_robotframework_nitpicky = True" would be optimal.

On the plus side: I now have a workaround to generate all screenshots with latest phantomjs as well as Firefox.

[datakurre]

@polyester It also doesn't generate any usual test logs. My reasoning was it would not be necessary, because it's possible to run the actual robot framework test runner against the docs to get "hard" errors and test logs: bin/pybot path_to_tested_rst_file (or bin/robot ... if bin/robot-server is running).

Does that make sense? Would that be enough? The good part here is that you can run it directly against the tested rst file.

Yet, there was an issue. The above is possible, because Robot Framework has support for running test from the RST-syntax we use with Sphinx. Yet, it only knows about Docutils and Sphinx does add a few custom directives, which caused errors about unknown directives in tested RST document. Therefore, plone.app.robotframework provides a custom bin/pybot, which registers those known missing Sphinx-directives so that pybot can be run against Sphinx documentation.

[datakurre]

https://github.com/plone/papyrus/blob/5.0/docs/robots.rst#how-to-work-on-a-single-document

[polyester]

Maybe I should better state the purpose: I want Travis to consider the build broken when one of the robotscreenshot-tests fails. I can't put Sphinx itself into -W mode, since indeed it generates a lot of warnings that are not really vital, but if one of the tests run by robotframework says "Fail" I want travis to consider the build broken, to give an indication that the last push has messed up the screenshots somehow.

Maybe an after-script grep on the log could do the trick.

[datakurre]

@polyester Ok. I've used to run pybot at Travis at first and then Sphinx when the tests pass. Probably, because you don't really need Travis generated Sphinx docs, we could only run pybot (test mode) at Travis :)

[datakurre]

I'll try to make a pull for this for docs repo.

[svx]

In a later state you do want to be able to run to run sphinx on travis in nitpicky -> http://sphinx-doc.org/config.html#confval-nitpicky together with -W and spell check to catch as many errors as possible.

I do that now on a couple of projects and it works really well :)

[svx]

What is the status here ? :)

[datakurre]

Thanks for reminding! I've probably forgotten this. Nitpick mode requires hard error support. Not too hard. Just need to schedule this :)

[datakurre]

@svx @polyester sphinxcontrib-robotframework >= 0.6.0 now generates both log files and hard errors with nitpicky-mode.

[svx]

@datakurre wow ! Thanks a lot ! !!!!

[svx]

OK this was forgotten, closing this now