Unit tests failing on numpy <= 1.8

[dkirkby]

The relevant test is:

    x, y = altaz_to_focalplane(angle, angle[:, np.newaxis],
        angle[:, np.newaxis, np.newaxis], 0.)
    assert x.shape == y.shape and x.shape == (3, 3, 3)

which works correctly with numpy 1.9 but gives a shape (3, 3, 1) for numpy <= 1.8. I am only seeing this error in TravisCL logs since I am running numpy 1.9.

[dkirkby]

Install a conda environment with numpy 1.8 for testing:

conda create --name np18 python=2.7 numpy=1.8 astropy scipy pyyaml

Try to reproduce this error locally:

source activate np18
py.test specsim

This fails because I forgot to include pytest in my environment, so fix that now:

conda install pytest

The pytest now fails on a different test because my np18 environment uses astropy 0.4.1, which I guess is the one that is intended to match numpy 1.8?

The current astropy is supposed to be backwards compatible to numpy 1.6 so try upgrading astropy by hand:

conda install astropy=1.0

This fails because conda's db says that astropy 1.0 requires numpy 1.10 and that the last version compatible with 1.8 is indeed 0.4.1:

conda info astropy
...
astropy 0.4.1 np18py26_0
------------------------
...
dependencies:
    argparse
    numpy 1.8*
    python 2.6*

Try forcing conda to update astropy and ignore the dependency error:

conda install astropy=1.0 --force

This fails even worse than before, so revert back to astropy 0.4.1:

conda install astropy=0.4.1

I can now finally reproduce the test using:

py.test specsim/tests/test_transform.py

For the record, here is my final test conda environment (using conda list -e):

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
astropy=0.4.1=np19py27_0
numpy=1.8.2=py27_0
openssl=1.0.2d=0
pip=7.1.2=py27_0
py=1.4.30=py27_0
pytest=2.8.1=py27_0
python=2.7.10=2
pyyaml=3.11=py27_1
readline=6.2=2
scipy=0.14.0=np18py27_0
setuptools=18.4=py27_0
sqlite=3.8.4.1=1
tk=8.5.18=0
wheel=0.26.0=py27_1
yaml=0.1.6=0
zlib=1.2.8=0

[dkirkby]

The source of the problem is a change in np.einsum for the following args:

v = np.einsum('ij...,j...->i...', R, u)

where

R.shape = (3, 3, 3, 1, 1)
u.shape = (3, 3, 3)

In numpy 1.9, the output has:

v.shape = (3, 3, 3, 3)

but in numpy 1.8:

v.shape = (3, 3, 3, 1)

The underlying broadcast rules have not changed, since this test gives the same result with 1.8 and 1.9:

R, u  = np.arange(3)[:,np.newaxis,np.newaxis], np.arange(9).reshape(3,3)
R.shape, u.shape, np.broadcast(R, u).shape
((3, 1, 1), (3, 3), (3, 3, 3))

I can reproduce the unit test failure with:

R, u  = np.arange(27).reshape(3,3,3)[:,:,:,np.newaxis,np.newaxis], np.arange(27).reshape(3,3,3)
np.einsum('ij...,j...->i...', R, u).shape

which gives (3, 3, 3, 1) in 1.8 and (3, 3, 3, 3) in 1.9.

[dkirkby]

I could replace the np.einsum call with some equivalent unrolled numpy calls, but since this is an edge case and the change would be less readable (and likely slower), instead detect this problem and raise a RuntimeError:

    output_shape = np.broadcast(alt, az, alt0, az0).shape
    ...
    if v[0].shape != output_shape:
        raise RuntimeError(
            'np.einsum does not broadcast correctly in numpy {}.'
            .format(np.version.version))

Update the unit test to pass if this exception is raised, so it can still be run in the more recent versions of numpy that are most relevant.