search

breathe-doc / breathe

ReStructuredText and Sphinx bridge to Doxygen
https://breathe-doc.org
Other
749 stars 198 forks source link

Investigate snapshot tests for output #733

Open michaeljones opened 3 years ago

michaeljones commented 3 years ago

Does anyone have any experience with snapshot testing with Python and maybe Sphinx?

I'm not sure what Sphinx manages tests. I'm familiar with snapshot testing from front end web development where it is sometimes not the best idea but I think it might be a good fit for our needs.

We might be able to set up something a bit manual with many tiny Sphinx projects testing individual scenarios against pre-generated and agreed upon output. It wouldn't be massively different to the 'compare' script we have now but it would be more automated and mean that the intentional changes to the output are recorded as changing snapshots in the PRs.

jakobandersen commented 3 years ago

Not really, but it sounds like a really good idea.

In Sphinx there are many different types of tests. Some are testing on the internal level (e.g., the C++ parsing tests) and some build to, e.g., html, and regex match certain parts. Some of the more recents tests also assert that the generated docutils trees are as expected.

But all the Breathe tests that are in documentation/source/specific.rst I don't think any of those types of tests are realistically possible. The resulting docutils trees are rather, so I basically agree with your idea. Though, instead of comparing HTML output it may be better to compare the XML output as that directly corresponds to the docutils nodes we anyway need to go back to in the code.

A potential issue is that even bug-fix version changes in Sphinx or docutils may change the output, so we should probably be prepared to have the multiple snapshots when we encounter these cases.

michaeljones commented 3 years ago

Thanks for sharing. Yeah, good idea to focus on the docutils trees instead of the html.

Might not go anywhere soon but might be worth investigating.