search

ContinuumIO / anaconda-issues

Anaconda issue tracking
646 stars 220 forks source link

Current versions of scipy and statsmodels do not play well #8979

Open AllenDowney opened 6 years ago

AllenDowney commented 6 years ago

It looks like the current versions of scipy and statsmodels are

scipy 1.0.0 py36hbf646e7_0
statsmodels 0.8.0 py36h8533d0b_0

With this combination, we have the problem described here:

https://github.com/statsmodels/statsmodels/issues/3931

statsmodels 0.8 uses chisqprob, which is no longer in scipy.

Apparently statsmodels 0.9 has resolved this issue, so this problem should be resolved by the next update of statsmodels. Any idea when that will be?

Thanks!

Anaconda or Miniconda version:
Operating System: Linux
conda info
     active environment : None
       user config file : /home/downey/.condarc
 populated config files : /home/downey/.condarc
          conda version : 4.5.0
    conda-build version : not installed
         python version : 3.6.4.final.0
       base environment : /home/downey/anaconda3  (writable)
           channel URLs : https://conda.anaconda.org/dsdale24/linux-64
                          https://conda.anaconda.org/dsdale24/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /home/downey/anaconda3/pkgs
                          /home/downey/.conda/pkgs
       envs directories : /home/downey/anaconda3/envs
                          /home/downey/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.0 requests/2.18.4 CPython/3.6.4 Linux/3.19.0-32-generic ubuntu/14.04 glibc/2.19
                UID:GID : 1000:1000
             netrc file : /home/downey/.netrc
           offline mode : False
conda list --show-channel-urls
# packages in environment at /home/downey/anaconda3:
#
# Name                    Version                   Build  Channel
_license                  1.1                      py36_1    defaults
alabaster                 0.7.10           py36h306e16b_0    defaults
anaconda                  5.1.0                    py36_2    defaults
anaconda-client           1.6.9                    py36_0    defaults
anaconda-navigator        1.6.2                    py36_0    defaults
anaconda-project          0.8.2            py36h44fb852_0    defaults
asn1crypto                0.24.0                   py36_0    defaults
astroid                   1.6.1                    py36_0    defaults
astropy                   2.0.3            py36h14c3975_0    defaults
attrs                     17.4.0                   py36_0    defaults
autobahn                  17.6.2                   py36_0    vpython
babel                     2.5.3                    py36_0    defaults
backports                 1.0              py36hfa02d7e_1    defaults
backports.shutil_get_terminal_size 1.0.0            py36hfea85ff_2    defaults
beautifulsoup4            4.6.0            py36h49b8c8c_1    defaults
bitarray                  0.8.1            py36h14c3975_1    defaults
bkcharts                  0.2              py36h735825a_0    defaults
blaze                     0.11.3           py36h4e06776_0    defaults
bleach                    2.1.2                    py36_0    defaults
bokeh                     0.12.13          py36h2f9c1c0_0    defaults
boto                      2.48.0           py36h6e4cd66_1    defaults
bottleneck                1.2.1            py36haac1ea0_0    defaults
bzip2                     1.0.6                h9a117a8_4    defaults
ca-certificates           2017.08.26           h1d4fec5_0    defaults
cairo                     1.14.12              h77bcde2_0    defaults
certifi                   2018.1.18                py36_0    defaults
cffi                      1.11.4           py36h9745a5d_0    defaults
chardet                   3.0.4            py36h0f667ec_1    defaults
click                     6.7              py36h5253387_0    defaults
cloudpickle               0.5.2                    py36_1    defaults
clyent                    1.2.2            py36h7e57e65_1    defaults
colorama                  0.3.9            py36h489cec4_0    defaults
conda                     4.5.0                    py36_0    defaults
conda-env                 2.6.0                h36134e3_1    defaults
contextlib2               0.5.5            py36h6c84a62_0    defaults
cryptography              2.1.4            py36hd09be54_0    defaults
curl                      7.58.0               h84994c4_0    defaults
cycler                    0.10.0           py36h93f1223_0    defaults
cython                    0.27.3           py36h1860423_0    defaults
cytoolz                   0.9.0            py36h14c3975_0    defaults
dask                      0.16.1                   py36_0    defaults
dask-core                 0.16.1                   py36_0    defaults
datashape                 0.5.4            py36h3ad6b5c_0    defaults
dbus                      1.12.2               hc3f9b76_1    defaults
decorator                 4.2.1                    py36_0    defaults
distributed               1.20.2                   py36_0    defaults
docutils                  0.14             py36hb0f60f5_0    defaults
entrypoints               0.2.3            py36h1aec115_2    defaults
et_xmlfile                1.0.1            py36hd6bccc3_0    defaults
expat                     2.2.5                he0dffb1_0    defaults
fastcache                 1.0.2            py36h14c3975_2    defaults
filelock                  2.0.13           py36h646ffb5_0    defaults
flask                     0.12.2           py36hb24657c_0    defaults
flask-cors                3.0.3            py36h2d857d3_0    defaults
fontconfig                2.12.4               h88586e7_1    defaults
freetype                  2.8                  hab7d2ae_1    defaults
get_terminal_size         1.0.0                haa9412d_0    defaults
gevent                    1.2.2            py36h2fe25dc_0    defaults
glib                      2.53.6               h5d9569c_2    defaults
glob2                     0.6              py36he249c77_0    defaults
gmp                       6.1.2                h6c8ec71_1    defaults
gmpy2                     2.0.8            py36hc8893dd_2    defaults
graphite2                 1.3.10               hf63cedd_1    defaults
greenlet                  0.4.12           py36h2d503a6_0    defaults
gst-plugins-base          1.12.4               h33fb286_0    defaults
gstreamer                 1.12.4               hb53b477_0    defaults
h5py                      2.7.1            py36h3585f63_0    defaults
harfbuzz                  1.7.4                hc5b324e_0    defaults
hdf5                      1.10.1               h9caa474_1    defaults
heapdict                  1.0.0                    py36_2    defaults
html5lib                  1.0.1            py36h2f9c1c0_0    defaults
icu                       58.2                 h9c2bf20_1    defaults
idna                      2.6              py36h82fb2a8_1    defaults
imageio                   2.2.0            py36he555465_0    defaults
imagesize                 0.7.1            py36h52d8127_0    defaults
impyla                    0.14.0                    <pip>
intel-openmp              2018.0.0             hc7b2577_8    defaults
ipykernel                 4.8.0                    py36_0    defaults
ipython                   6.2.1            py36h88c514a_1    defaults
ipython_genutils          0.2.0            py36hb52b0d5_0    defaults
ipywidgets                7.1.1                    py36_0    defaults
isort                     4.2.15           py36had401c0_0    defaults
itsdangerous              0.24             py36h93cc618_1    defaults
jbig                      2.1                  hdba287a_0    defaults
jdcal                     1.3              py36h4c697fb_0    defaults
jedi                      0.11.1                   py36_0    defaults
jinja2                    2.10             py36ha16c418_0    defaults
jpeg                      9b                   h024ee3a_2    defaults
jsonschema                2.6.0            py36h006f8b5_0    defaults
jupyter                   1.0.0                    py36_4    defaults
jupyter_client            5.2.2                    py36_0    defaults
jupyter_console           5.2.0            py36he59e554_1    defaults
jupyter_core              4.4.0            py36h7c827e3_0    defaults
jupyterlab                0.31.5                   py36_0    defaults
jupyterlab_launcher       0.10.2                   py36_0    defaults
lazy-object-proxy         1.3.1            py36h10fcdad_0    defaults
libcurl                   7.58.0               h1ad7b7a_0    defaults
libedit                   3.1                  heed3624_0    defaults
libffi                    3.2.1                hd88cf55_4    defaults
libgcc                    4.8.5                         2    defaults
libgcc-ng                 7.2.0                h7cc24e2_2    defaults
libgfortran               3.0.0                         1    defaults
libgfortran-ng            7.2.0                h9f7466a_2    defaults
libiconv                  1.14                          0    defaults
libpng                    1.6.34               hb9fc6fc_0    defaults
libsodium                 1.0.15               hf101ebd_0    defaults
libssh2                   1.8.0                h9cfc8f7_4    defaults
libstdcxx-ng              7.2.0                h7a57d05_2    defaults
libtiff                   4.0.9                h28f6b97_0    defaults
libtool                   2.4.6                h544aabb_3    defaults
libxcb                    1.12                 hcd93eb1_4    defaults
libxml2                   2.9.7                h26e45fe_0    defaults
libxslt                   1.1.32               h1312cb7_0    defaults
llvmlite                  0.21.0           py36ha241eea_0    defaults
locket                    0.2.0            py36h787c0ad_1    defaults
lxml                      4.1.1            py36hf71bdeb_1    defaults
lzo                       2.10                 h49e0be7_2    defaults
markupsafe                1.0              py36hd9260cd_1    defaults
matplotlib                2.1.2            py36h0e671d2_0    defaults
mccabe                    0.6.1            py36h5ad9710_1    defaults
mistune                   0.8.3                    py36_0    defaults
mkl                       2018.0.1             h19d6760_4    defaults
mkl-service               1.1.2            py36h17a0993_4    defaults
mpc                       1.0.3                hec55b23_5    defaults
mpfr                      3.1.5                h11a74b3_2    defaults
mpmath                    1.0.0            py36hfeacd6b_2    defaults
msgpack-python            0.5.1            py36h6bb024c_0    defaults
multipledispatch          0.4.9            py36h41da3fb_0    defaults
navigator-updater         0.1.0                    py36_0    defaults
nbconvert                 5.3.1            py36hb41ffb7_0    defaults
nbformat                  4.4.0            py36h31c9010_0    defaults
ncurses                   6.0                  h9df7e31_2    defaults
networkx                  2.1                      py36_0    defaults
nltk                      3.2.5            py36h7532b22_0    defaults
nose                      1.3.7            py36hcdf7029_2    defaults
notebook                  5.4.0                    py36_0    defaults
numba                     0.36.2          np114py36hc6662d5_0    defaults
numexpr                   2.6.4            py36hc4a3f9a_0    defaults
numpy                     1.14.0           py36h3dfced4_1    defaults
numpydoc                  0.7.0            py36h18f165f_0    defaults
odo                       0.5.1            py36h90ed295_0    defaults
olefile                   0.45.1                   py36_0    defaults
openpyxl                  2.4.10                   py36_0    defaults
openssl                   1.0.2n               hb7f436b_0    defaults
packaging                 16.8             py36ha668100_1    defaults
palettable                3.1.0                     <pip>
pandas                    0.22.0           py36hf484d3e_0    defaults
pandoc                    1.19.2.1             hea2e7c5_1    defaults
pandocfilters             1.4.2            py36ha6701b7_1    defaults
pango                     1.41.0               hd475d92_0    defaults
parso                     0.1.1            py36h35f843b_0    defaults
partd                     0.3.8            py36h36fd896_0    defaults
patchelf                  0.9                  hf79760b_2    defaults
path.py                   10.5             py36h55ceabb_0    defaults
pathlib2                  2.3.0            py36h49efa8e_0    defaults
patsy                     0.5.0                    py36_0    defaults
pcre                      8.41                 hc27e229_1    defaults
pep8                      1.7.1                    py36_0    defaults
pexpect                   4.3.1                    py36_0    defaults
pickleshare               0.7.4            py36h63277f8_0    defaults
pillow                    5.0.0            py36h3deb7b8_0    defaults
pint                      0.7.2                    py36_2    unidata
pip                       9.0.1            py36h6c6f9ce_4    defaults
pixman                    0.34.0               hceecf20_3    defaults
pkginfo                   1.4.1            py36h215d178_1    defaults
pluggy                    0.6.0            py36hb689045_0    defaults
ply                       3.10             py36hed35086_0    defaults
prompt_toolkit            1.0.15           py36h17d85b1_0    defaults
psutil                    5.4.3            py36h14c3975_0    defaults
ptyprocess                0.5.2            py36h69acd42_0    defaults
py                        1.5.2            py36h29bf505_0    defaults
pycodestyle               2.3.1            py36hf609f19_0    defaults
pycosat                   0.6.3            py36h0a5515d_0    defaults
pycparser                 2.18             py36hf9f622e_1    defaults
pycrypto                  2.6.1            py36h14c3975_7    defaults
pycurl                    7.43.0.1         py36hb7f436b_0    defaults
pyflakes                  1.6.0            py36h7bd6a15_0    defaults
pygments                  2.2.0            py36h0d3125c_0    defaults
pylint                    1.8.2                    py36_0    defaults
pyodbc                    4.0.22           py36hf484d3e_0    defaults
pyopenssl                 17.5.0           py36h20ba746_0    defaults
pyparsing                 2.2.0            py36hee85983_1    defaults
pyqt                      5.6.0            py36h0386399_5    defaults
pysocks                   1.6.7            py36hd97a5b1_1    defaults
pytables                  3.4.2            py36h3b5282a_2    defaults
pytest                    3.3.2                    py36_0    defaults
python                    3.6.4                hc3d631a_1    defaults
python-dateutil           2.6.1            py36h88d3b88_1    defaults
pytz                      2017.3           py36h63b9c63_0    defaults
pywavelets                0.5.2            py36he602eb0_0    defaults
pyyaml                    3.12             py36hafb9ca4_1    defaults
pyzmq                     16.0.3           py36he2533c7_0    defaults
qt                        5.6.2               h974d657_12    defaults
qtawesome                 0.4.4            py36h609ed8c_0    defaults
qtconsole                 4.3.1            py36h8f73b5b_0    defaults
qtpy                      1.3.1            py36h3691cc8_0    defaults
readline                  7.0                  ha6073c6_4    defaults
redis                     2.10.6                    <pip>
requests                  2.18.4           py36he2e5f8d_1    defaults
rope                      0.10.7           py36h147e2ec_0    defaults
ruamel_yaml               0.15.35          py36h14c3975_1    defaults
scikit-image              0.13.1           py36h14c3975_1    defaults
scikit-learn              0.19.1           py36h7aa7ec6_0    defaults
scipy                     1.0.0            py36hbf646e7_0    defaults
seaborn                   0.8.1            py36hfad7ec4_0    defaults
send2trash                1.4.2                    py36_0    defaults
setuptools                38.4.0                   py36_0    defaults
simplegeneric             0.8.1                    py36_2    defaults
singledispatch            3.4.0.3          py36h7a266c3_0    defaults
sip                       4.18.1           py36h51ed4ed_2    defaults
six                       1.11.0           py36h372c433_1    defaults
snowballstemmer           1.2.1            py36h6febd40_0    defaults
sortedcollections         0.5.3            py36h3c761f9_0    defaults
sortedcontainers          1.5.9                    py36_0    defaults
sphinx                    1.6.6                    py36_0    defaults
sphinxcontrib             1.0              py36h6d0f590_1    defaults
sphinxcontrib-websupport  1.0.1            py36hb5cb234_1    defaults
spyder                    3.2.6                    py36_0    defaults
sqlalchemy                1.2.1            py36h14c3975_0    defaults
sqlite                    3.22.0               h1bed415_0    defaults
statsmodels               0.8.0            py36h8533d0b_0    defaults
sympy                     1.1.1            py36hc6d1c1c_0    defaults
tblib                     1.3.2            py36h34cf8b6_0    defaults
terminado                 0.8.1                    py36_1    defaults
testpath                  0.3.1            py36h8cadb63_0    defaults
thriftpy                  0.3.9                     <pip>
tk                        8.6.7                hc745277_3    defaults
toolz                     0.9.0                    py36_0    defaults
tornado                   4.5.3                    py36_0    defaults
traitlets                 4.3.2            py36h674d592_0    defaults
txaio                     2.8.0                    py36_0    vpython
typing                    3.6.2            py36h7da032a_0    defaults
ujson                     1.35                     py36_0    defaults
unicodecsv                0.14.1           py36ha668878_0    defaults
unixodbc                  2.3.4                hc36303a_1    defaults
urllib3                   1.22             py36hbe7ace6_0    defaults
vpnotebook                0.1.3                    py36_1    vpython
vpython                   7.4.2                    py36_0    vpython
wcwidth                   0.1.7            py36hdf4376a_0    defaults
webencodings              0.5.1            py36h800622e_1    defaults
werkzeug                  0.14.1                   py36_0    defaults
wheel                     0.30.0           py36hfd4bba0_1    defaults
widgetsnbextension        3.1.0                    py36_0    defaults
wrapt                     1.10.11          py36h28b7045_0    defaults
xlrd                      1.1.0            py36h1db9f0c_1    defaults
xlsxwriter                1.0.2            py36h3de1aca_0    defaults
xlwt                      1.3.0            py36h7b00a1f_0    defaults
xz                        5.2.3                h55aa19d_2    defaults
yaml                      0.1.7                had09818_2    defaults
zeromq                    4.2.2                hbedb6e5_2    defaults
zict                      0.1.3            py36h3a3bf81_0    defaults
zlib                      1.2.11               ha838bed_2    defaults
msarahan commented 6 years ago

In general the best approach is to bump the conda-forge feedstock. The CI there helps iron out any recipe problems, and we can pull in changes very directly now. You can also submit PRs to the recipes at AnacondaRecipes: https://github.com/AnacondaRecipes/statsmodels-feedstock, but submitting it to conda-forge is preferable, thanks to their CI setup. We will see to this update.

AllenDowney commented 6 years ago

Mike, thanks for the quick response. I don't understand most of what you just said. Is this something that will be fixed soon, or are you telling me how to work around it?

On Fri, Mar 23, 2018 at 10:46 AM, Mike Sarahan notifications@github.com wrote:

In general the best approach is to bump the conda-forge feedstock. The CI there helps iron out any recipe problems, and we can pull in changes very directly now. You can also submit PRs to the recipes at AnacondaRecipes: https://github.com/AnacondaRecipes/statsmodels-feedstock, but submitting it to conda-forge is preferable, thanks to their CI setup. We will see to this update.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/8979#issuecomment-375688437, or mute the thread https://github.com/notifications/unsubscribe-auth/ABy37RAM-Sjvd5X8aWvQfG4QBpGHvFuwks5thQqsgaJpZM4S3HIr .

msarahan commented 6 years ago

Sorry, just looked at this, and it seems like statsmodels 0.9 is not released yet. We can't update packages until then. We can hotfix the index metadata to introduce a constraint so that statsmodels 0.8 depends on scipy <1.0.

msarahan commented 6 years ago

The Anaconda packages are produced by recipes. There used to be a private internal pool of these. Now they're all public, and they are all drawn from conda-forge, which is a community organization for building packages. They created some clever automation that takes a lot of the burden out of the trial and error of package building. What I pointed you to is two places where you can edit the recipe (to do the update). With conda-forge, you'd submit a PR to their https://github.com/conda-forge/statsmodels-feedstock repo, and once that PR is merged, your new packages would appear on the conda-forge channel automatically.

With those packages proven, we can pull in the recipe changes to Anaconda, and build them on our build system, which we keep separate for the sake of security guarantees for our customers.

AllenDowney commented 6 years ago

Thanks for the explanation!

On Fri, Mar 23, 2018 at 10:54 AM, Mike Sarahan notifications@github.com wrote:

The Anaconda packages are produced by recipes. There used to be a private internal pool of these. Now they're all public, and they are all drawn from conda-forge, which is a community organization for building packages. They created some clever automation that takes a lot of the burden out of the trial and error of package building. What I pointed you to is two places where you can edit the recipe (to do the update). With conda-forge, you'd submit a PR to their https://github.com/conda-forge/statsmodels-feedstock repo, and once that PR is merged, your new packages would appear on the conda-forge channel automatically.

With those packages proven, we can pull in the recipe changes to Anaconda, and build them on our build system, which we keep separate for the sake of security guarantees for our customers.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/8979#issuecomment-375691102, or mute the thread https://github.com/notifications/unsubscribe-auth/ABy37XXyavsCJ1Cq6dApYIKsqzTMCyfCks5thQyfgaJpZM4S3HIr .