search

probcomp / Venturecxx

Primary implementation of the Venture probabilistic programming system
http://probcomp.csail.mit.edu/venture/
GNU General Public License v3.0
28 stars 6 forks source link

Tag (especially slow) non-backend-agnostic tests with @on_backend #499

Open axch opened 8 years ago

axch commented 8 years ago

to prevent them from being redundantly rerun by e.g. the all-crashes script (that Jenkins uses).

Issue: Not sure how to mechanize maintenance of the desired state (which, of course, is why this weeding seems necessary).

axch commented 8 years ago

Project management decision: Is this a priority?

raxraxraxraxrax commented 8 years ago

It makes sense that we'd want to run backend-agnostic tests on multiple backends, but running slow ones on only one backend most of the time also makes sense. I don't think it's a high priority since if I understand correctly it's not leading to major resource crunch, test failures, or missing information; we're just using more compute than we need to?

axch commented 8 years ago

Any of those that have a non-negligible false fail rate impose an additional cost, namely increasing the false fail rate of the test suite as a whole, thereby reducing our trust in it. In the glorious future the proposing of which is #549, this will manifest as needing somewhat more computation for every probabilistic test in the whole test suite in order to retain the same level of trust.