search

holoviz-topics / EarthSim

Tools for working with and visualizing environmental simulations.
https://earthsim.holoviz.org
BSD 3-Clause "New" or "Revised" License
65 stars 21 forks source link

erdc channel's "stevedore" clobbers "six" #11

Closed ceball closed 6 years ago

ceball commented 6 years ago

Installing erdc's stevedore appears to clobber six with its own bundled version. The version of six included by stevedore is not up to date enough for e.g. matplotlib, so we get problems like this:

screen shot 2018-02-15 at 12 57 54 am screen shot 2018-02-15 at 12 29 05 am 
$ wget https://anaconda.org/erdc/stevedore/1.6.0/download/noarch/stevedore-1.6.0-py_0.tar.bz2
$ tar -tf stevedore-1.6.0-py_0.tar.bz2 
Scripts/.stevedore-pre-link.bat
bin/.stevedore-pre-link.sh
info/files
info/index.json
info/recipe.json
info/recipe/bld.bat
info/recipe/build.sh
info/recipe/meta.yaml
link.py
site-packages/argparse-1.3.0.dist-info/DESCRIPTION.rst
site-packages/argparse-1.3.0.dist-info/METADATA
site-packages/argparse-1.3.0.dist-info/RECORD
site-packages/argparse-1.3.0.dist-info/WHEEL
site-packages/argparse-1.3.0.dist-info/metadata.json
site-packages/argparse-1.3.0.dist-info/top_level.txt
site-packages/argparse.py
site-packages/six-1.9.0.dist-info/DESCRIPTION.rst
site-packages/six-1.9.0.dist-info/METADATA
site-packages/six-1.9.0.dist-info/RECORD
site-packages/six-1.9.0.dist-info/WHEEL
site-packages/six-1.9.0.dist-info/metadata.json
site-packages/six-1.9.0.dist-info/top_level.txt
site-packages/six.py
site-packages/stevedore-1.6.0-py2.7.egg-info/PKG-INFO
site-packages/stevedore-1.6.0-py2.7.egg-info/SOURCES.txt
site-packages/stevedore-1.6.0-py2.7.egg-info/dependency_links.txt
site-packages/stevedore-1.6.0-py2.7.egg-info/entry_points.txt
site-packages/stevedore-1.6.0-py2.7.egg-info/not-zip-safe
site-packages/stevedore-1.6.0-py2.7.egg-info/pbr.json
site-packages/stevedore-1.6.0-py2.7.egg-info/requires.txt
site-packages/stevedore-1.6.0-py2.7.egg-info/top_level.txt
site-packages/stevedore/__init__.py
site-packages/stevedore/dispatch.py
site-packages/stevedore/driver.py
site-packages/stevedore/enabled.py
site-packages/stevedore/example/__init__.py
site-packages/stevedore/example/base.py
site-packages/stevedore/example/fields.py
site-packages/stevedore/example/load_as_driver.py
site-packages/stevedore/example/load_as_extension.py
site-packages/stevedore/example/setup.py
site-packages/stevedore/example/simple.py
site-packages/stevedore/extension.py
site-packages/stevedore/hook.py
site-packages/stevedore/named.py
site-packages/stevedore/sphinxext.py
site-packages/stevedore/tests/__init__.py
site-packages/stevedore/tests/extension_unimportable.py
site-packages/stevedore/tests/manager.py
site-packages/stevedore/tests/test_callback.py
site-packages/stevedore/tests/test_dispatch.py
site-packages/stevedore/tests/test_driver.py
site-packages/stevedore/tests/test_enabled.py
site-packages/stevedore/tests/test_example_fields.py
site-packages/stevedore/tests/test_example_simple.py
site-packages/stevedore/tests/test_extension.py
site-packages/stevedore/tests/test_hook.py
site-packages/stevedore/tests/test_named.py
site-packages/stevedore/tests/test_sphinxext.py
site-packages/stevedore/tests/test_test_manager.py
site-packages/stevedore/tests/utils.py
jbednar commented 6 years ago

@dharhas, is there any reason stevedore bundles six and argparse instead of just depending on them? That's an unusual arrangement and to me seems safe only if it bundled them as submodules within its own module, not at the top level where there will be conflicts with other packages.

ceball commented 6 years ago

The packaging of stevedore on conda-forge looks less problematic, so maybe it would be enough to just pin stevedore's version to be higher than the stevedore on erdc's channel. I've done that in #10; please tell me if that's going to cause a problem (with quest?) I'm unaware of.

If everyone were using conda 4.4, we could instead specify to install only gssha, filigree, libfiligree from the erdc channel.

dharhas commented 6 years ago

I've removed stevedore from the erdc channel. I think it is an artifact of a time before conda-forge was available. I'm a little confused why a lower version of stevedore is preferred over a higher version but I guess channel has priority? Our policy is to only maintain packages in erdc that cannot be put in an community channel like conda-forge.

dharhas commented 6 years ago

@jbednar no idea. Stevedore and pbr (python build reasonableness) are part of openshift and I've noticed that they have their own particular way of doing things that don't always match the wider community. Back in the day I picked stevedore as a framework for a plugin architecture in quest and pbr to deal with versioning. If I had to do it over, I would probably go with versioneer for versions and I'm not sure what else is out there for plugins.

ceball commented 6 years ago

I'm a little confused why a lower version of stevedore is preferred over a higher version but I guess channel has priority?

I think so, yes.

jbednar commented 6 years ago

In that case, presumably we could have avoided the issue by bumping the erdc channel to be down below conda-forge?

ceball commented 6 years ago

Depends. It was the first thing I considered, but I took the conservative option of avoiding only the one package known to cause a problem.

By 'depends', I mean: (a) is there any package already on conda-forge that's also on erdc, where we need the package to come from erdc? (b) what if in the future someone puts one of the must-come-from-erdc packages onto conda-forge?

However, Dharhas has clarified that the "policy is to only maintain packages in erdc that cannot be put in an community channel like conda-forge", so maybe the erdc channel should be put at a lower priority than conda-forge, and we should just see what happens?

And also, maybe various packages should be removed from the erdc channel that are on conda-forge? There appear to be quite a few. However, I don't know why those packages exist - there might be a good reason.

jbednar commented 6 years ago

I think we should pin versions very specifically as the final deliverable from this phase, and then relax them when we resume development in the next phase. So at this point it shouldn't matter; there shouldn't be any significant ambiguity. Eventually, my guess is to put the erdc channel lower in priority, because new versions of things are likely to appear on conda-forge, and we'll then realize that and be forced to adapt accordingly.

dharhas commented 6 years ago

In general, the erdc channel is only for stuff not available in other channels, so it should probably be last in priority.

ceball commented 6 years ago

I think we should pin versions very specifically

@jbednar, so just to be clear, I should pin all packages in environment.yml to exact versions currently being used?

And then later on, we'll also make a conda package where we don't do such pinning.

jbednar commented 6 years ago

If you're volunteering, then yes, please do! I was just saying abstractly that someone should do it. :-) We should make a note (e.g. a comment on that line of the file) for any of those pins that are actually known to be required, so that we don't mistakenly delete those later when we remove the rest of the pinning.

dharhas commented 6 years ago

And also, maybe various packages should be removed from the erdc channel that are on conda-forge? There appear to be quite a few. However, I don't know why those packages exist - there might be a good reason.

There are some packages in the erdc channel that were created by the hashdist/conda portion of this project in 2017. The reason as far as I understand, is that they have compilation options suited for hpc that were not accepted by conda-forge. Those need to stay. None are packages EarthSim uses. Any older ones can be removed if causing problems.

ceball commented 6 years ago

Thanks for explaining that.

Anyway, the original issue is resolved.