search

mperrin / webbpsf

James Webb Space Telescope PSF simulation tool - NOTE THIS VERSION OF REPO IS SUPERCEDED BY spacetelescope/webbpsf
BSD 3-Clause "New" or "Revised" License
16 stars 15 forks source link

conda install webbpsf followed by conda install notebook downgrades WebbPSF #127

Closed josePhoenix closed 8 years ago

josePhoenix commented 8 years ago

It appears that AstroConda does not provide a WebbPSF 0.5 package that is compatible with the version of the Jupyter Notebook available from Continuum Analytics. It's not clear whether this should be fixed by our Science Software Branch with a package stating compatibility with NumPy 1.10 (cc @jhunkeler) or by Continuum Analytics rebuilding the notebook package with the latest NumPy.

¯¯_(ツ)_/¯¯

To reproduce:

  1. conda create -n somename stsci - install main AstroConda metapackage
  2. Verify WebbPSF 0.5.0 is installed
  3. conda install notebook - install the Jupyter Notebook (packaged by Continuum)
  4. Verify that WebbPSF 0.4.0 is now the installed version

Full log of replication below:

00:28:53 tanuki:~ jlong$ conda create -n testcasewebbpsfdowngrade stsci
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ........
Solving package specifications: .........

Package plan for installation in environment /Users/jlong/anaconda/envs/testcasewebbpsfdowngrade:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pandas-0.18.1              |      np111py35_0         7.1 MB
    pyfftw-0.9.2               |      np111py35_1          88 KB
    scikit-learn-0.17.1        |      np111py35_2         3.6 MB
    stsci.ndimage-0.10.1       |      np111py35_0         149 KB
    stsci.tools-3.4.1          |      np111py35_0         411 KB
    calcos-3.1.8               |      np111py35_0         368 KB
    fitsblender-0.2.6          |      np111py35_0          61 KB
    nictools-1.1.3             |      np111py35_0         109 KB
    pyqtgraph-0.9.11           |      np111py35_1         934 KB
    stginga-0.1.1              |      np111py35_0          87 KB
    stistools-1.1              |      np111py35_0          68 KB
    stsci.image-2.2.0          |      np111py35_0          22 KB
    stsci.imagemanip-1.1.2     |      np111py35_0          10 KB
    stsci.numdisplay-1.6.1     |      np111py35_0          41 KB
    stwcs-1.2.3                |      np111py35_0         176 KB
    wfpc2tools-1.0.3           |      np111py35_0          30 KB
    astroimtools-0.1           |      np111py35_0          35 KB
    costools-1.2.1             |      np111py35_0          40 KB
    imexam-0.5.2               |      np111py35_0         335 KB
    poppy-0.5.0                |      np111py35_0         232 KB
    pysynphot-0.9.8.3          |      np111py35_0         4.4 MB
    scikit-image-0.12.3        |      np111py35_1        17.8 MB
    specview-0.1               |      np111py35_0         134 KB
    stsci.sphere-0.2           |      np111py35_1         333 KB
    wfc3tools-1.3.1            |      np111py35_0         3.5 MB
    acstools-2.0.2             |      np111py35_0          55 KB
    photutils-0.2.1            |      np111py35_1         256 KB
    reftools-1.7.1             |      np111py35_1         113 KB
    webbpsf-0.5.0              |      np111py35_0         169 KB
    drizzlepac-2.1.4           |      np111py35_0         716 KB
    cube-tools-0.0.0           |      np111py35_0          59 KB
    asv-0.1.1                  |      np111py35_0         857 KB
    stsci-data-analysis-1.0.0  |      np111py35_2          951 B
    stsci-hst-1.0.3            |      np111py35_0           1 KB
    stsci-1.0.0                |      np111py35_1           1 KB
    ------------------------------------------------------------
                                           Total:        42.1 MB

The following NEW packages will be INSTALLED:

    acstools:            2.0.2-np111py35_0
    alabaster:           0.7.8-py35_0
    appdirs:             1.4.0-py35_0
    appnope:             0.1.0-py35_0
    asdf:                1.0.3-np111py35_0
    astroimtools:        0.1-np111py35_0
    astrolib.coords:     0.39.6-np111py35_0
    astropy:             1.2.1-np111py35_0
    asv:                 0.1.1-np111py35_0
    babel:               2.3.3-py35_0
    backports:           1.0-py35_0
    calcos:              3.1.8-np111py35_0
    cfitsio:             3.370-1
    costools:            1.2.1-np111py35_0
    crds:                7.0.1-np111py35_0
    cube-tools:          0.0.0-np111py35_0
    cycler:              0.10.0-py35_0
    cython:              0.24-py35_0
    d2to1:               0.2.12-py35_0
    decorator:           4.0.10-py35_0
    docutils:            0.12-py35_2
    drizzlepac:          2.1.4-np111py35_0
    ds9:                 7.4-0
    fftw:                3.3.4-0
    fitsblender:         0.2.6-np111py35_0
    freetype:            2.5.5-1
    get_terminal_size:   1.0.0-py35_0
    ginga:               2.5.20160706100500-np111py35_0
    glueviz:             0.7.2-py35_0
    h5py:                2.6.0-np111py35_1
    hdf5:                1.8.16-0
    hstcal:              1.0.1-0
    htc_utils:           0.1-py35_0
    imagesize:           0.7.1-py35_0
    imexam:              0.5.2-np111py35_0
    ipykernel:           4.3.1-py35_0
    ipython:             4.2.0-py35_1
    ipython_genutils:    0.1.0-py35_0
    jbig:                2.1-0
    jinja2:              2.8-py35_1
    jpeg:                8d-1
    jsonschema:          2.5.1-py35_0
    jupyter_client:      4.3.0-py35_0
    jupyter_core:        4.1.0-py35_0
    libgcc:              4.8.5-1
    libpng:              1.6.22-0
    libtiff:             4.0.6-2
    markupsafe:          0.23-py35_2
    matplotlib:          1.5.1-np111py35_0
    mkl:                 11.3.3-0
    networkx:            1.11-py35_0
    nictools:            1.1.3-np111py35_0
    nose:                1.3.7-py35_1
    numpy:               1.11.1-py35_0
    numpydoc:            0.5-py35_1
    openssl:             1.0.2h-1
    pandas:              0.18.1-np111py35_0
    path.py:             8.2.1-py35_0
    pexpect:             4.0.1-py35_0
    photutils:           0.2.1-np111py35_1
    pickleshare:         0.7.2-py35_0
    pillow:              3.3.0-py35_0
    pip:                 8.1.2-py35_0
    pkgconfig:           1.1.0-py35_0
    poppy:               0.5.0-np111py35_0
    ptyprocess:          0.5.1-py35_0
    purge_path:          1.0.0-py35_0
    py:                  1.4.31-py35_0
    pyds9:               1.8.1-np111py35_0
    pyfftw:              0.9.2-np111py35_1
    pygments:            2.1.3-py35_0
    pyopengl:            3.1.1a1-np111py35_0
    pyparsing:           2.1.4-py35_0
    pyqt:                4.11.4-py35_3
    pyqtgraph:           0.9.11-np111py35_1
    pyregion:            1.1.2-np111py35_0
    pysynphot:           0.9.8.3-np111py35_0
    pytest:              2.9.2-py35_0
    python:              3.5.1-5
    python-dateutil:     2.5.3-py35_0
    python.app:          1.2-py35_4
    pytools:             2016.1-np111py35_1
    pytz:                2016.4-py35_0
    pyyaml:              3.11-py35_4
    pyzmq:               15.2.0-py35_1
    qt:                  4.8.7-3
    qtconsole:           4.2.1-py35_0
    readline:            6.2-2
    reftools:            1.7.1-np111py35_1
    scikit-image:        0.12.3-np111py35_1
    scikit-learn:        0.17.1-np111py35_2
    scipy:               0.17.1-np111py35_1
    setuptools:          23.0.0-py35_0
    simplegeneric:       0.8.1-py35_1
    sip:                 4.16.9-py35_0
    six:                 1.10.0-py35_0
    snowballstemmer:     1.2.1-py35_0
    specview:            0.1-np111py35_0
    sphinx:              1.4.1-py35_0
    sphinx_rtd_theme:    0.1.9-py35_0
    sqlite:              3.13.0-0
    stginga:             0.1.1-np111py35_0
    stistools:           1.1-np111py35_0
    stsci:               1.0.0-np111py35_1
    stsci-data-analysis: 1.0.0-np111py35_2
    stsci-hst:           1.0.3-np111py35_0
    stsci.convolve:      2.1.3-np111py35_0
    stsci.distutils:     0.3.8-np111py35_0
    stsci.image:         2.2.0-np111py35_0
    stsci.imagemanip:    1.1.2-np111py35_0
    stsci.imagestats:    1.4.1-np111py35_0
    stsci.ndimage:       0.10.1-np111py35_0
    stsci.numdisplay:    1.6.1-np111py35_0
    stsci.skypac:        0.9-py35_0
    stsci.sphere:        0.2-np111py35_1
    stsci.sphinxext:     1.2.2-py35_0
    stsci.stimage:       0.2.1-np111py35_0
    stsci.tools:         3.4.1-np111py35_0
    stwcs:               1.2.3-np111py35_0
    tk:                  8.5.18-0
    tornado:             4.3-py35_1
    traitlets:           4.2.1-py35_0
    webbpsf:             0.5.0-np111py35_0
    webbpsf-data:        0.5.0-0
    wfc3tools:           1.3.1-np111py35_0
    wfpc2tools:          1.0.3-np111py35_0
    wheel:               0.29.0-py35_0
    xlrd:                1.0.0-py35_0
    xpa:                 2.1.17-0
    xz:                  5.2.2-0
    yaml:                0.1.6-0
    zlib:                1.2.8-3

Proceed ([y]/n)?

Fetching packages ...
pandas-0.18.1- 100% |################################| Time: 0:00:01   6.90 MB/s
pyfftw-0.9.2-n 100% |################################| Time: 0:00:00   1.07 MB/s
scikit-learn-0 100% |################################| Time: 0:00:00   5.19 MB/s
stsci.ndimage- 100% |################################| Time: 0:00:00   1.20 MB/s
stsci.tools-3. 100% |################################| Time: 0:00:00   2.01 MB/s
calcos-3.1.8-n 100% |################################| Time: 0:00:00   1.93 MB/s
fitsblender-0. 100% |################################| Time: 0:00:00   7.39 MB/s
nictools-1.1.3 100% |################################| Time: 0:00:00   1.22 MB/s
pyqtgraph-0.9. 100% |################################| Time: 0:00:00   3.46 MB/s
stginga-0.1.1- 100% |################################| Time: 0:00:00   1.02 MB/s
stistools-1.1- 100% |################################| Time: 0:00:00 842.14 kB/s
stsci.image-2. 100% |################################| Time: 0:00:00   7.25 MB/s
stsci.imageman 100% |################################| Time: 0:00:00   4.58 MB/s
stsci.numdispl 100% |################################| Time: 0:00:00   8.56 MB/s
stwcs-1.2.3-np 100% |################################| Time: 0:00:00   1.28 MB/s
wfpc2tools-1.0 100% |################################| Time: 0:00:00   9.26 MB/s
astroimtools-0 100% |################################| Time: 0:00:00   8.68 MB/s
costools-1.2.1 100% |################################| Time: 0:00:00   7.99 MB/s
imexam-0.5.2-n 100% |################################| Time: 0:00:00   1.90 MB/s
poppy-0.5.0-np 100% |################################| Time: 0:00:00   1.49 MB/s
pysynphot-0.9. 100% |################################| Time: 0:00:00   7.95 MB/s
scikit-image-0 100% |################################| Time: 0:00:01  10.58 MB/s
specview-0.1-n 100% |################################| Time: 0:00:00   1.25 MB/s
stsci.sphere-0 100% |################################| Time: 0:00:00   1.93 MB/s
wfc3tools-1.3. 100% |################################| Time: 0:00:00   7.72 MB/s
acstools-2.0.2 100% |################################| Time: 0:00:00  10.99 MB/s
photutils-0.2. 100% |################################| Time: 0:00:00   1.60 MB/s
reftools-1.7.1 100% |################################| Time: 0:00:00   1.37 MB/s
webbpsf-0.5.0- 100% |################################| Time: 0:00:00   1.15 MB/s
drizzlepac-2.1 100% |################################| Time: 0:00:00   3.15 MB/s
cube-tools-0.0 100% |################################| Time: 0:00:00  11.14 MB/s
asv-0.1.1-np11 100% |################################| Time: 0:00:00   3.42 MB/s
stsci-data-ana 100% |################################| Time: 0:00:00 747.66 kB/s
stsci-hst-1.0. 100% |################################| Time: 0:00:00   1.11 MB/s
stsci-1.0.0-np 100% |################################| Time: 0:00:00 853.44 kB/s
Extracting packages ...
[      COMPLETE      ]|#######################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|#######################################################################################| 100%
#
# To activate this environment, use:
# $ source activate testcasewebbpsfdowngrade
#
# To deactivate this environment, use:
# $ source deactivate
#
00:31:07 tanuki:~ jlong$ source activate testcasewebbpsfdowngrade
discarding /Users/jlong/anaconda/bin from PATH
prepending /Users/jlong/anaconda/envs/testcasewebbpsfdowngrade/bin to PATH
(testcasewebbpsfdowngrade)00:31:47 tanuki:~ jlong$ conda install notebook
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ........
Solving package specifications: .........

Package plan for installation in environment /Users/jlong/anaconda/envs/testcasewebbpsfdowngrade:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    python-3.5.2               |                0        12.6 MB
    scipy-0.17.1               |      np110py35_1        11.9 MB
    pyfftw-0.9.2               |      np110py35_1          88 KB
    scikit-learn-0.17.1        |      np110py35_2         3.6 MB
    scikit-image-0.12.3        |      np110py35_1        17.8 MB
    ------------------------------------------------------------
                                           Total:        46.0 MB

The following NEW packages will be INSTALLED:

    entrypoints:         0.2.2-py35_0
    mistune:             0.7.2-py35_1
    nbconvert:           4.2.0-py35_0
    nbformat:            4.0.1-py35_0
    notebook:            4.2.1-py35_0
    pywcs:               1.12.1-np110py35_0
    terminado:           0.6-py35_0

The following packages will be UPDATED:

    astroimtools:        0.1-np111py35_0                --> 0.1-np110py35_0
    astrolib.coords:     0.39.6-np111py35_0             --> 0.39.6-np110py35_0
    asv:                 0.1.1-np111py35_0              --> 0.1.1-np110py35_0
    costools:            1.2.1-np111py35_0              --> 1.2.1-np110py35_0
    cube-tools:          0.0.0-np111py35_0              --> 0.0.0-np110py35_0
    fitsblender:         0.2.6-np111py35_0              --> 0.2.6-np110py35_0
    h5py:                2.6.0-np111py35_1              --> 2.6.0-np110py35_1
    imexam:              0.5.2-np111py35_0              --> 0.5.2-np110py35_0
    matplotlib:          1.5.1-np111py35_0              --> 1.5.1-np110py35_0
    nictools:            1.1.3-np111py35_0              --> 1.1.3-np110py35_0
    pandas:              0.18.1-np111py35_0             --> 0.18.1-np110py35_0
    photutils:           0.2.1-np111py35_1              --> 0.2.1-np110py35_1
    pyds9:               1.8.1-np111py35_0              --> 1.8.1-np110py35_0
    pyfftw:              0.9.2-np111py35_1              --> 0.9.2-np110py35_1
    pyopengl:            3.1.1a1-np111py35_0            --> 3.1.1a1-np110py35_0
    pyregion:            1.1.2-np111py35_0              --> 1.1.2-np110py35_0
    python:              3.5.1-5                        --> 3.5.2-0
    scikit-image:        0.12.3-np111py35_1             --> 0.12.3-np110py35_1
    scikit-learn:        0.17.1-np111py35_2             --> 0.17.1-np110py35_2
    scipy:               0.17.1-np111py35_1             --> 0.17.1-np110py35_1
    specview:            0.1-np111py35_0                --> 0.1-np110py35_0
    stistools:           1.1-np111py35_0                --> 1.1-np110py35_0
    stsci.convolve:      2.1.3-np111py35_0              --> 2.1.3-np110py35_0
    stsci.distutils:     0.3.8-np111py35_0              --> 0.3.8-np110py35_0
    stsci.image:         2.2.0-np111py35_0              --> 2.2.0-np110py35_0
    stsci.imagemanip:    1.1.2-np111py35_0              --> 1.1.2-np110py35_0
    stsci.imagestats:    1.4.1-np111py35_0              --> 1.4.1-np110py35_0
    stsci.ndimage:       0.10.1-np111py35_0             --> 0.10.1-np110py35_0
    stsci.numdisplay:    1.6.1-np111py35_0              --> 1.6.1-np110py35_0
    stsci.stimage:       0.2.1-np111py35_0              --> 0.2.1-np110py35_0
    stsci.tools:         3.4.1-np111py35_0              --> 3.4.1-np110py35_0
    stwcs:               1.2.3-np111py35_0              --> 1.2.3-np110py35_0
    wfc3tools:           1.3.1-np111py35_0              --> 1.3.1-np110py35_0
    wfpc2tools:          1.0.3-np111py35_0              --> 1.0.3-np110py35_0

The following packages will be DOWNGRADED:

    acstools:            2.0.2-np111py35_0              --> 2.0.0-np110py35_0
    asdf:                1.0.3-np111py35_0              --> 1.0.2-np110py35_0
    astropy:             1.2.1-np111py35_0              --> 1.1.2-np110py35_0
    calcos:              3.1.8-np111py35_0              --> 3.1.7-np110py35_0
    crds:                7.0.1-np111py35_0              --> 6.0.0-np110py35_0
    drizzlepac:          2.1.4-np111py35_0              --> 2.1.3-np110py35_0
    ginga:               2.5.20160706100500-np111py35_0 --> 2.5.20151215011852-np110py35_0
    hstcal:              1.0.1-0                        --> 1.0.0-0
    numpy:               1.11.1-py35_0                  --> 1.10.4-py35_2
    poppy:               0.5.0-np111py35_0              --> 0.4.0-np110py35_0
    pyqtgraph:           0.9.11-np111py35_1             --> 0.9.11-np110py35_0
    pysynphot:           0.9.8.3-np111py35_0            --> 0.9.8.2-np110py35_0
    pytools:             2016.1-np111py35_1             --> 2016.1-py35_0
    reftools:            1.7.1-np111py35_1              --> 1.7.1-np110py35_0
    stginga:             0.1.1-np111py35_0              --> 0.0.0-np110py35_0
    stsci:               1.0.0-np111py35_1              --> 1.0.0-np110py35_0
    stsci-data-analysis: 1.0.0-np111py35_2              --> 1.0.0-np110py35_0
    stsci-hst:           1.0.3-np111py35_0              --> 1.0.1-np110py35_0
    stsci.sphere:        0.2-np111py35_1                --> 0.2-np110py35_0
    webbpsf:             0.5.0-np111py35_0              --> 0.4.0-np110py35_0
    webbpsf-data:        0.5.0-0                        --> 0.4.0-0

Proceed ([y]/n)?

Fetching packages ...
python-3.5.2-0 100% |####################################################################| Time: 0:00:01  11.32 MB/s
scipy-0.17.1-n 100% |####################################################################| Time: 0:00:01   9.04 MB/s
pyfftw-0.9.2-n 100% |####################################################################| Time: 0:00:00   1.04 MB/s
scikit-learn-0 100% |####################################################################| Time: 0:00:00   5.62 MB/s
scikit-image-0 100% |####################################################################| Time: 0:00:01  10.46 MB/s
Extracting packages ...
[      COMPLETE      ]|#######################################################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|#######################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|#######################################################################################| 100%
(testcasewebbpsfdowngrade)00:32:26 tanuki:~ jlong$
mperrin commented 8 years ago

I don't understand - what requirements are forcing this downgrade?

I think it's not anything in webbpsf or poppy themselves, but has to be something in the astroconda-contrib specifications. The requirements in poppy are just 'numpy>=1.8.0' so any version above that should be OK. Right? But conversely I don't see what's wrong with the meta.yaml files for either in astroconda.

josePhoenix commented 8 years ago

I think it's even more ridiculous. I think it might be that Continuum Analytics has not built a notebook package with the appropriate magic bits that say it works with NumPy 1.11. @jhunkeler and I talked it over and iirc that's what we came up with.

josePhoenix commented 8 years ago

Conversely, I suppose, we have not built a webbpsf/webbpsf-data/poppy package set for the new version that says it works with 1.10. That should be within our capabilities to fix.

mperrin commented 8 years ago

But that's what confuses me. I don't see anything in https://github.com/astroconda/astroconda-contrib/blob/master/poppy/meta.yaml which indicates that it's not compatible with numpy 1.10. Same for https://github.com/astroconda/astroconda-contrib/blob/master/webbpsf/meta.yaml

jhunkeler commented 8 years ago

The blame is split between Continuum and myself. I stopped building against numpy 1.10 in favor of 1.11 to cut down on the overall build-time. It is Continuum's fault for simply not keeping their packages up to date with the latest numpy.

Why did I go from 1.10 to 1.11 so suddenly? At the time, Continuum compiled Astropy 1.2.1 against numpy 1.11, but not 1.10 and some of our software already relied on features in Astropy 1.2.1.

Tough situation -- poor execution, I'll admit.

In any case, I posted a message on the Anaconda mailing list when this happened. It took a few days (a few days too long), but they ended up recompiling Astropy 1.2.1 against numpy >=1.8. We should probably just tell them about notebook... there's high probability they have no idea it's even missing 1.11 support.

Scratch everything I just said... pywcs was deprecated in stsci-hst 1.0.3 release, and thus was not compiled against numpy 1.11.

jhunkeler commented 8 years ago

@josePhoenix @mperrin

Ok, it's definitely Monday.

When I upgraded the base packages to 1.11 the stsci metapackage received a build bump instead of a version bump. So instead of cascading the numpy 1.11 packages it continued to provide numpy 1.10.

I suggest running conda update stsci then conda install notebook if you want to hop up to 1.11. Since notebook has no hard dependency on numpy, installing it by itself will not automatically upgrade stsci. So if you try again, your conda client should go out and check the dependencies and be able to resolve notebook without downgrading.

Hopefully this change clears up the mess. Let me know what happens on your end.

josePhoenix commented 8 years ago

Thanks @jhunkeler ! Just built a new astroconda env with conda create --name astroconda python=3.5 stsci notebook and successfully imported WebbPSF v0.5.0.