Why is jenkins not failing on #93 and #94

[gregory-marton]

Edit: Links to #93 and #94

[gregory-marton]

The difference between my machine and the jenkins box, where these pass, is that on my machine, matplotlib.version is '1.5.0', whereas on probcomp-4 it is '1.4.3'. This is another case where pip, by failing to update to latest during pip install, if you happen already to have an adequate version, makes it impossible for us to do configuration management.

In an ideal world, we might support both versions for a while. If we update to the 1.5.0, I wonder if it will fail in the 1.4.3 environment.

[gregory-marton]

Can we keep testing both somehow? Should we bother, in general? Should we be installing everything from scratch for each run, to ensure that we're still working with the latest and greatest? What if a client doesn't want to work with the latest and greatest for some of their code, and what of the cost to us of always staying up to date?

[gregory-marton]

Can we test both? Yes: set up pip-stable and pip-head jenkins jobs, the former triggering the latter, where pip-head somehow programmatically undoes all the stability-specs in setup.py, and does a virtualenv build. This still keeps all the non-pip dependencies at stable, but that's fine.

Should we? My bias is to do at-pip-head testing only manually, on something like a quarterly upgrade cycle, as described above. This is a request for comments. For the class, I think ensuring stable deps is trump.

[gregory-marton]

Added check.sh to the pip-install jenkins, which is at-pip-head, and rerunning now. Should fail.

[gregory-marton]

Good. Pip-install failed ... but on cython rather than informatively as expected. Looking into that.

[axch]

Re: builds with different dependency sets: I have been pretty happy with Docker as an isolation environment for testing Venture without depending on what's installed on the build machine. The setup is in https://probcomp-3.csail.mit.edu/job/venture-sdist-docker-pip-install/ . In theory the scriptage implementing this could be adjusted to admit specifying particular versions of dependencies via e.g. a requirements.txt file with == version constraints, so one could have as many test environments as desired.

Warning: the Docker pip install build takes around half an hour, because pip recompiles numpy, scipy, and pandas inside the container every time.

Other warning: Docker will cache the results of apt-get upgrade, so the Debian packages will be at some specific version (until one clears the cache, in which case they will get upgraded at the next build).

[axch]

My generic $0.02 on the question of range of versions to support.

I think the first goal should be to make sure that pip install bayeslite always produces a working install. In principle, that means testing all combinations of dependency versions that are permitted by the dependencies specified in setup.py. In practice, it should suffice to test a "representative" set; something like "all top, all bottom, all top except one at bottom, all bottom except one at top, a few random other configurations." [Disclaimer: Venture is not tested anywhere near so much; rather it's "top" and "whatever happened to be on probcomp-3, which is not top". I was thinking of revisiting that for the mid-January release, but I don't know whether I will get to it.]

I think the second goal should be to make the range of dependencies permitted by setup.py as broad as practical, to maximize the chances of Bayeslite slotting into an existing environment without forcing people to change what they have installed.

I think the third goal should be to make sure that setup.py never requires users to downgrade.

I think the fourth goal should be to make sure that setup.py never requires users to upgrade past what's packaged for Ubuntu LTS, but I doubt that goal is actually achievable (at least for Venture). (Don't know what the corresponding standard is for Mac, if any.)

I don't actually have enough context on the specifics of the case you are dealing with now to suggest a solution to this constraint satisfaction problem.

[gregory-marton]

So as per previous experience, one really does have to

pip install cython numpy scipy

before doing

pip install bdbcontrib

for the latter to succeed. I wonder if putting those into setup_requires rather than just ... oh dear, scipy isn't even specified. It's required transitively by sklearn though. But it still failed without. Hmm...

Anyway, so if I put them into setup_requires, then they'll get installed first, but they'll get installed by easy_install, rather than by pip. That may be fine. https://pip.pypa.io/en/stable/reference/pip_install/#controlling-setup-requires

[axch]

General suggestion: Avoid easy_install like the plague.

Also, as @riastradh-probcomp discovered, everything in setup_requires will be installed for any use of setup.py, including checking the version number. I recall him being quite frustrated by this; I do not recall whether he came up with a best practice for not having the problem.

[gregory-marton]

@axch adds:

Debatable: easy_install silently screws up the user's machine in ways that require nontrivial investigation to undo. For my part, I would prefer a two-line installation instruction that explicitly mentions

 pip install cython

(and whatever else) as a prerequisite. Crosscat had this problem; is there a solution there? Also, pystan presumably has this problem; we could choose to use them as a proxy for acceptable behavior of entrants into the field.

https://pystan.readthedocs.org/en/latest/getting_started.html

(Which may end up with easy_install; I haven't checked).

[gregory-marton]

The problem for cython and numpy is indeed inherited from crosscat, and is not well solved there. Looking into pystan...

[gregory-marton]

Okay, after significant modifications (adding testing, etc.), the pip-install jenkins now fails with the same messages as indicated in #93 and #94.

Here is the process at this time:

set -eux

venv_dir=`mktemp -dt "jenkins-pip-install.XXXXXXXXXX"`
virtualenv $venv_dir
. $venv_dir/bin/activate
# Install these first, because crosscat's setup.py fails if these aren't present beforehand:
pip install --upgrade cython numpy scipy
# Separately install bayeslite-apsw because it just doesn't work without the install options, 
# and having "bayeslite-apsw>=3.8.0" in bayeslite's install_requires doesn't cut it.
pip install --upgrade bayeslite-apsw --install-option="fetch" --install-option="--sqlite" --install-option=" version=3.9.2"
# Now we can install bdbcontrib (which will install crosscat and bayeslite) and runipy
pip install --upgrade bdbcontrib runipy
mkdir $venv_dir/bdb-demo
cd $venv_dir/bdb-demo
$venv_dir/bin/bayesdb-demo fetch
runipy ./Satellites.ipynb --matplotlib --stdout | tee $venv_dir/satellites-output.txt

# TODO: somehow check the output?

pip install --upgrade mock pytest # tests_require
pip install --upgrade pillow --global-option="build_ext" --global-option="--disable-jpeg" # tests_require
cd $WORKSPACE/../crosscat-tests/
./check.sh
cd $WORKSPACE/../bayeslite-master-crashes/
./check.sh tests
cd $WORKSPACE/../bdbcontrib-tests/
./check.sh tests
cd $WORKSPACE
/bin/rm -fr $venv_dir

[riastradh-probcomp]

Date: Tue, 22 Dec 2015 15:30:21 -0800 From: Gregory Marton notifications@github.com

Can we keep testing both somehow? Should we bother, in general? Should we be installing everything from scratch for each run, to ensure that we're still working with the latest and greatest? What if a client doesn't want to work with the latest and greatest for some of their code, and what of the cost to us of always staying up to date?

We should pick the oldest version we're interested in dealing with (my usual first pick: whatever is in the latest Ubuntu LTS), and test everything against that; and also have a `pip install' test to make sure whatever pip will install by default works.

If I recall correctly, that is what we do for bayeslite with sqlite3/apsw right now on jenkins: most jobs use apsw 3.8.0, and there's one pip-install job that installs the most recent version of everything on pypi.

[riastradh-probcomp]

Using setup_requires in setup.py entails easy_install, so we don't want to do that.