bitprophet
commented
8 years ago Heck, I just ran into some wasted time tonight because of the mismatch between the fixture-generating methods & a real changelog. Real changelog sufficed for troubleshooting until implementation worked, but test continued to fail because it didn't actually match the changelog. And this is not the first time this has happened.
Right now the test suite is largely hand-made, semi-internal data structures, plus various wrappers tying them into real-enough structures that the top level API calls can work with. So e.g. one instantiates
IssueandReleaseobjects (or shorthand creating them) in some order, it round-trips into a parsed set of changelog nodes, then that result is introspected and asserted about.It's pretty thorny, involves a large amount of (kinda ugly) extra test code, many layers due to differing abstraction needs for different tests, and most importantly means test fixtures look nothing like real (or doc example) changelogs - and it'd be nice if that were the case.
IIRC this was done in part because actually rendering Sphinx/ReST text without a real set of doc files on the filesystem was impossible or very hard (and of course, truly touching the FS makes tests that much slower, as per the integration suite).
That was 2-3 years ago and I'm wondering if A) my memory is wrong or B) things have improved since then (Sphinx has seen a good speed of dev in the last year+).
If it's still not feasible to do so (or if I am misremembering re: how often we generate things programmatically in a way that a text changelog simply can't handle) we should at least examine tightening up all the generation/manipulation code so it's more consistent & easier to reason about.