Are these downgraded packages from conda-forge ok?

[rsignell-usgs]

@ocefpaf, when I just built the new pangeo.esipfed notebook container, I got this message:

The following packages will be DOWNGRADED:

    bleach:             2.1.3-py_0                            conda-forge           --> 1.5.0-py36_0     conda-forge
    html5lib:           1.0.1-py_0                            conda-forge           --> 0.9999999-py36_0 conda-forge

Does that mean those conda-forge packages need upgrading?

[ocefpaf]

Not really b/c the newer versions exist. I need to know in what step that happens to investigate if that is a conda solver issue or a new dependency that is being installed that has pinned packages and causing the downgrade.

[ocefpaf]

This env is solvable:

bleach==2.1.3
html5lib==1.0.1
conda-forge/label/dev::rasterio
cython
dask-kubernetes
dask-tensorflow
erddapy
fusepy
geoviews
gridgeo
lz4
nb_conda_kernels
nbserverproxy
opencv
palettable
pydensecrf
python-blosc
s3fs
siphon
tensorflow
tensorflow-hub
utide
websocket-client
zarr

Which means that the downgrade there is b/c conda's solver is confused and cannot optimize the package stack properly. I'm investigating if there are old builds that are picked up, if so I can remove them and force conda to get only the latest builds.

[ocefpaf]

@rsignell-usgs this is what I feared. In the pinned vs the unpinned solver diff we can see that the pinned installs 5 different packages (absl-py, astor, gast, grpcio, markdown) versus 2 (mock and pbr) in the unpinned version (packages that only changed version are not counted).

Note that the dependencies do change and more or less packages is expected. The problem is that conda prefers the easy path, i.e. less packages, rather than the updated packages.

Pinging @msarahan here who can shed more light on this issue.

--- pinned.txt  2018-06-18 17:36:45.192456622 -0300
+++ unpinned.txt    2018-06-18 17:42:57.348989938 -0300
+absl-py                   0.2.2                      py_0    conda-forge
+astor                     0.6.2                      py_0    conda-forge
-bleach                    2.1.3                      py_0    conda-forge
+bleach                    1.5.0                    py36_0    conda-forge
+gast                      0.2.0                      py_0    conda-forge
+grpcio                    1.12.0           py36hdbcaa40_0    defaults
-html5lib                  1.0.1                      py_0    conda-forge
+html5lib                  0.9999999                py36_0    conda-forge
+markdown                  2.6.11                     py_0    conda-forge
-mock                      2.0.0                    py36_0    conda-forge
-pbr                       4.0.4                      py_0    conda-forge
-tensorflow                1.1.0                    py36_0    conda-forge
+tensorboard               1.8.0                    py36_1    conda-forge
+tensorflow                1.8.0                    py36_1    conda-forge
+termcolor                 1.1.0                    py36_1    conda-forge

For now, if you need updated version of html5lib and bleach, I recommend to add them explicitly in the file and pin the version you need.

[rsignell-usgs]

@msarahan, pretty confusing if conda doesn't pick the updated packages, no?

[msarahan]

It does seem unintuitive, but conda is especially maximizing the versions of explicitly named dependencies: https://github.com/conda/conda/blob/master/conda/resolve.py#L840

maximizing other dependencies that are not explicitly named comes much later, after things like overall package count: https://github.com/conda/conda/blob/master/conda/resolve.py#L877

[rsignell-usgs]

So previously I was thinking that I should minimize the number of packages I specify in the environment.yml, but now I'm thinking that I should maximize the packages I specify, to make sure I get the most recent packages that are compatible. @msarahan , is that true?

[ocefpaf]

but conda is especially maximizing the versions of explicitly named dependencies

tensorflow is the only explicitly listed dependency in that dance and it is downgraded. Not sure if that policy is working as expected then.

[ocefpaf]

Ack, no! tensorflow 1.1.0 is in the pinned version :unamused:

Let me see investigate why that is happening.

[ocefpaf]

@rsignell-usgs tensorflow 1.8.0 requires tensorboard and tensorboard pins html5lib ==0.9999999.

In that case conda is correct and there is nothing we can do about it. (Unless that pin is wrong but I'm not a maintainer of tensorboard.)

[rsignell-usgs]

Ah, I see: https://github.com/tensorflow/tensorboard/commit/e90374757959ea85cffe040dbe8193406b0a28fd

Okay, so tensorboard needs those older packages. Closing. Thanks for the insight!