REGRESS tests should be integration-only

probcomp / bayeslite

BayesDB on SQLite. A Bayesian database table for querying the probable implications of data as easily as SQL databases query the data itself.

http://probcomp.csail.mit.edu/software/bayesdb

Apache License 2.0

922 stars 64 forks source link

REGRESS tests should be integration-only #551

Closed riastradh-probcomp closed 7 years ago

riastradh-probcomp commented 7 years ago

[x] (defer to another day) Consider adopting the --integration mechanism from cgpm.
[x] (3ce2d94c2a8f262ca5b606c73d6d818158d4b152) Mark REGRESS tests as integration tests.
[x] Add a bayeslite-master-ubuntu1604-integration job to jenkins.

fsaad commented 7 years ago

The --integration mechanism could be used to replace the existing __ci_slow convention.

We probably should explicitly mark --integration with a reason, for example __ci_slow versus __ci_dependency, since integration tests might be slow, or might require optional dependencies, etc.

fsaad commented 7 years ago

While addressing the issue of integration tests and dependencies, we should also address this pattern for skipping tests based on dependencies from tests/test_loom#L33-L37:

def loom_analyze(csv_filename):
    try:
        import loom
    except ImportError:
        pytest.skip('no loom')
        return

...

def test_loom_dha__ci_slow():
    loom_analyze(dha_csv)

def test_loom_satellites__ci_slow():
    loom_analyze(satellites_csv)

riastradh-probcomp commented 7 years ago

My idea with --integration was to mark that as the one for integration tests requiring a full suite of dependencies; we could separately have --slow, --network, &c., and maybe --all to run them all.