Gain confidence that our dependency version bounds are correct

[axch]

Ideally, any system that has all our listed dependencies installed, with versions agreeing with all of our version range limits, should be able to use all of Venture's functionality successfully. Actually testing exactly that is combinatorially difficult, but a selected subset may be useful.

Suggested path:

• Get an end-to-end script that takes a dependency spec, builds a system with those dependencies present, and runs the acceptance test (e.g., verifyinstall.sh)
  • Can start with the "virtualenv on Ubuntu (no site packages)" isolation level
• Make a Jenkins build that runs that script on many configurations:
  • all latest
  • all earliest
  • all but one latest, that one earliest
  • all but one earliest, that one latest
  • choose a valid configuration at random
  • can afford to run the full version of this quite rarely, e.g. daily or weekly
  • do we want to mess with varying version numbers of our dependencies' dependencies?
  • the transitive dependency set can be discovered by running pip freeze after an install into a clean virtualenv
• Doesn't matter at first whether the deduction of test configurations from the listed dependencies is manual or automatic; can automate it when manual maintenance becomes a hassle
• It may be possible (but may not be necessary) to reuse some installation work by keeping the virtualenv between configurations and selectively upgrading inside it.
  • One upgrade path will of course not suffice to hit all the desired configurations
  • This may not produce the same effect as building fresh, because of the versions of dependencies' dependencies

[axch]

This can presumably reuse the script(s) needed for #171

[axch]

Specifically, one obvious first step would be to generalize script/jenkins/check_sdist.sh to accept an optional requirements.txt file and install dependencies from that into the virtualenv before installing the Venture source distribution (they will not be auto-upgraded).

[axch]

Progress report:

• script/jenkins/venture-in-docker.sh and script/jenkins/check_sdist.sh have been generalized to accept a requirements file as an argument and pip install -r its contents before doing pip install venture. This permit controlling the versions of dependencies that are present on the system.
• There is a tool called tool/minimal_deps.py that reads our requirements declaration and emits a requirements file consisting of the oldest (yes, oldest) versions of those packages. This tool also serves as an example of how to work with setuptools requirements and versions objects.
• There is a Jenkins build, https://probcomp-3.csail.mit.edu/job/venture-sdist-docker-pip-dependency-variation/ , that tests Venture against the oldest versions of its dependencies that our listed dependencies nominally permit.

New issue: I am now skeptical of our test suite's coverage of our dependencies, because the above build passes even though we have four unbounded dependencies: dill, networkx, ipyparallel, and jsonschema. Also, ipython is listed as >=1.2 (current is 4 something), which does not strike me as believable.

Consequently, I declare this blocked on #183 (for actually touching our dependencies) and on #50 (for getting rid of our dependency on IPython).

[axch]

We might also want to test against minimal versions of our test dependencies (currently, the venture-sdist-docker-pip-dependency-variation build only controls the main dependencies). The way to do that would be:

• Extract the test_requires field of setup.py into a text file like install_requires.
• Generalize tool/minimum_deps.py to accept a source file name as an argument.
• Possibly adjust venture-in-docker and check_built_sdist to accept another file of requirements to install after capability checking but before running the test suite (or not, since "works without the test dependencies" is being tested elsewhere)
• Tweak the build to control the test dependencies also

[axch]

Status update: #50 is done, but this is still blocked on #183.