search

holoviz / datashader

Quickly and accurately render even the largest data.
http://datashader.org
BSD 3-Clause "New" or "Revised" License
3.32k stars 365 forks source link

Trouble running tests with provided conda env #724

Open ivirshup opened 5 years ago

ivirshup commented 5 years ago

I'm having trouble running the tests locally using the provided conda environment. The environment was created following the instructions in the readme: 

conda env create --name datashader --file ./examples/environment.yml
conda activate datashader
pip install -e .
pytest

pytest immediately fails with the following traceback:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/_pytest/main.py", line 206, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/_pytest/config/__init__.py", line 639, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pluggy/hooks.py", line 311, in call_historic
INTERNALERROR>     res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pluggy/manager.py", line 68, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pluggy/manager.py", line 62, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pytest_benchmark/plugin.py", line 1092, in pytest_configure
INTERNALERROR>     config._benchmarksession = BenchmarkSession(config)
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pytest_benchmark/plugin.py", line 568, in __init__
INTERNALERROR>     self.logger = Logger(self.verbose, config)
INTERNALERROR>   File "/Users/isaac/miniconda3/envs/datashader/lib/python3.6/site-packages/pytest_benchmark/plugin.py", line 519, in __init__
INTERNALERROR>     self.pytest_warn = config.warn
INTERNALERROR> AttributeError: 'Config' object has no attribute 'warn'

I was able to get tests partially running under a different environment which doesn't have pytest-benchmark installed.

This is using the latest master (3171d8869f63214cc161da755e88c30abe8b5098) with macOS v10.14.3.

Output of `conda list` ``` # packages in environment at /Users/isaac/miniconda3/envs/datashader: # # Name Version Build Channel affine 2.2.2 py_0 conda-forge antlr-python-runtime 4.7.2 py36_1000 conda-forge appnope 0.1.0 py36_1000 conda-forge asn1crypto 0.24.0 py36_1003 conda-forge atomicwrites 1.3.0 py_0 conda-forge attrs 19.1.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge beautifulsoup4 4.7.1 py36_1001 conda-forge blas 1.1 openblas conda-forge bleach 3.1.0 py_0 conda-forge blosc 1.16.3 h0a44026_0 conda-forge bokeh 1.0.4 py36_1000 conda-forge boost-cpp 1.68.0 h6f8c590_1000 conda-forge bzip2 1.0.6 h1de35cc_1002 conda-forge ca-certificates 2019.3.9 hecc5488_0 conda-forge cachey 0.1.1 pypi_0 pypi cairo 1.16.0 h9247486_1000 conda-forge cartopy 0.17.0 py36h929c6f0_1004 conda-forge certifi 2019.3.9 py36_0 conda-forge cf-units 2.1.1 py36h917ab60_1000 conda-forge cffi 1.12.2 py36h2d6ddff_1 conda-forge cftime 1.0.3.4 py36h917ab60_1000 conda-forge chardet 3.0.4 py36_1003 conda-forge click 7.0 py_0 conda-forge click-plugins 1.0.4 py_0 conda-forge cligj 0.5.0 py_0 conda-forge cloudpickle 0.8.0 py_0 conda-forge colorcet 1.0.0 py_0 conda-forge cryptography 2.5 py36hdbc3d79_1 conda-forge curl 7.64.0 heae2a1f_0 conda-forge cycler 0.10.0 py_1 conda-forge cytoolz 0.9.0.1 py36h1de35cc_1001 conda-forge dask 1.1.4 py_0 conda-forge dask-core 1.1.4 py_0 conda-forge datashader 0.7.0a1.post4+g3171d88 dev_0 datashape 0.5.4 py_1 conda-forge decorator 4.3.2 py_0 conda-forge defusedxml 0.5.0 py_1 conda-forge descartes 1.1.0 py_2 conda-forge dill 0.2.9 py36_0 conda-forge distributed 1.26.0 py36_1 conda-forge entrypoints 0.3 py36_1000 conda-forge expat 2.2.5 h0a44026_1002 conda-forge fastparquet 0.2.1 py36h917ab60_1001 conda-forge fiona 1.8.4 py36h0e3174d_1002 conda-forge flake8 3.7.7 py36_0 conda-forge fontconfig 2.13.1 h1e4e890_1000 conda-forge freetype 2.9.1 h597ad8a_1005 conda-forge freexl 1.0.5 h1de35cc_1002 conda-forge gdal 2.4.0 py36h0e3174d_1002 conda-forge geopandas 0.4.1 py_0 conda-forge geos 3.7.1 h0a44026_1000 conda-forge geotiff 1.4.3 hce09ea4_1000 conda-forge geoviews 1.6.2 py_0 conda-forge geoviews-core 1.6.2 py_0 conda-forge gettext 0.19.8.1 hcca000d_1001 conda-forge giflib 5.1.4 h1de35cc_1001 conda-forge glib 2.58.3 h2836805_1001 conda-forge graphviz 2.38.0 hc6cc99f_1011 conda-forge hdf4 4.2.13 hf3c6af0_1002 conda-forge hdf5 1.10.4 nompi_h646315f_1105 conda-forge heapdict 1.0.0 py36_1000 conda-forge holoviews 1.11.3 py_0 pyviz/label/dev icu 58.2 h0a44026_1000 conda-forge idna 2.8 py36_1000 conda-forge imageio 2.4.1 py36_1000 conda-forge ipykernel 5.1.0 py36h24bf2e0_1002 conda-forge ipython 7.1.1 py36h24bf2e0_1000 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.4.2 py_0 conda-forge iris 2.2.0 py36_1003 conda-forge jedi 0.13.3 py36_0 conda-forge jinja2 2.10 py_1 conda-forge jpeg 9c h1de35cc_1001 conda-forge json-c 0.13.1 h1de35cc_1001 conda-forge jsonschema 3.0.1 py36_0 conda-forge jupyter 1.0.0 py_1 conda-forge jupyter_client 5.2.4 py_3 conda-forge jupyter_console 6.0.0 py_0 conda-forge jupyter_core 4.4.0 py_0 conda-forge jupyter_dashboards 0.7.0 py36_1000 conda-forge kealib 1.4.10 hf5ed860_1002 conda-forge kiwisolver 1.0.1 py36h04f5b5a_1002 conda-forge krb5 1.16.3 h24a3359_1000 conda-forge libcurl 7.64.0 h76de61e_0 conda-forge libcxx 7.0.0 h2d50403_1 conda-forge libdap4 3.19.1 hae55d67_1000 conda-forge libedit 3.1.20170329 hcfe32e1_1001 conda-forge libffi 3.2.1 h0a44026_1005 conda-forge libgdal 2.4.0 h89caebc_1002 conda-forge libgfortran 3.0.1 0 conda-forge libiconv 1.15 h1de35cc_1004 conda-forge libkml 1.3.0 h71ee1b2_1009 conda-forge libnetcdf 4.6.2 h6b88ef6_1001 conda-forge libpng 1.6.36 ha441bb4_1000 conda-forge libpq 10.6 hbe1e24e_1000 conda-forge libsodium 1.0.16 h1de35cc_1001 conda-forge libspatialindex 1.8.5 h0a44026_1003 conda-forge libspatialite 4.3.0a h0cd9627_1026 conda-forge libssh2 1.8.0 hf30b1f0_1003 conda-forge libtiff 4.0.10 h79f4b77_1001 conda-forge libxml2 2.9.8 hf14e9c8_1005 conda-forge libxslt 1.1.32 h33a18ac_1002 conda-forge llvm-meta 7.0.0 0 conda-forge llvmlite 0.26.0 py36h3fea490_1000 conda-forge locket 0.2.0 py_2 conda-forge lxml 4.3.2 py36h7166777_0 conda-forge mapclassify 2.0.1 py_0 conda-forge markupsafe 1.1.1 py36h1de35cc_0 conda-forge marshmallow 2.19.0 pypi_0 pypi matplotlib 3.0.3 py36_0 conda-forge matplotlib-base 3.0.3 py36hf043ca5_0 conda-forge mccabe 0.6.1 py_1 conda-forge mistune 0.8.4 py36h1de35cc_1000 conda-forge more-itertools 4.3.0 py36_1000 conda-forge msgpack-python 0.6.1 py36h04f5b5a_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge munch 2.3.2 py_0 conda-forge nbconvert 5.4.1 py_2 conda-forge nbformat 4.4.0 py_1 conda-forge ncurses 6.1 h0a44026_1002 conda-forge netcdf4 1.4.3.2 py36hf8bc7f3_0 conda-forge networkx 2.2 py_1 conda-forge notebook 5.7.5 py36_0 conda-forge numba 0.41.0 py36h1702cab_1000 conda-forge numexpr 2.6.9 py36h1702cab_1000 conda-forge numpy 1.16.2 py36_blas_openblash486cb9f_0 [blas_openblas] conda-forge olefile 0.46 py_0 conda-forge openblas 0.3.3 hdc02c5d_1001 conda-forge openjpeg 2.3.0 h3bf0609_1003 conda-forge openssl 1.0.2r h1de35cc_0 conda-forge owslib 0.17.1 py_0 conda-forge packaging 19.0 py_0 conda-forge pandas 0.24.1 py36h0a44026_0 conda-forge pandoc 2.6 1 conda-forge pandocfilters 1.4.2 py_1 conda-forge param 1.8.2 py_0 conda-forge paramnb 2.0.4 py_0 conda-forge parso 0.3.4 py_0 conda-forge partd 0.3.9 py_0 conda-forge patsy 0.5.1 py_0 conda-forge pcre 8.41 h0a44026_1003 conda-forge pexpect 4.6.0 py36_1000 conda-forge pickleshare 0.7.5 py36_1000 conda-forge pillow 5.4.1 py36hbddbef0_1000 conda-forge pip 19.0.3 py36_0 conda-forge pixman 0.34.0 h1de35cc_1003 conda-forge pluggy 0.9.0 py_0 conda-forge poppler 0.67.0 hb974355_6 conda-forge poppler-data 0.4.9 1 conda-forge postgresql 10.6 ha1bbaa7_1000 conda-forge proj4 5.2.0 h1de35cc_1001 conda-forge prometheus_client 0.6.0 py_0 conda-forge prompt_toolkit 2.0.9 py_0 conda-forge psutil 5.6.0 py36h1de35cc_0 conda-forge ptyprocess 0.6.0 py36_1000 conda-forge py 1.8.0 py_0 conda-forge pycodestyle 2.5.0 py_0 conda-forge pycparser 2.19 py_0 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pyepsg 0.4.0 py_0 conda-forge pyflakes 2.1.1 py_0 conda-forge pygments 2.3.1 py_0 conda-forge pykdtree 1.3.1 py36h917ab60_1002 conda-forge pyke 1.1.1 py36_1000 conda-forge pyopenssl 19.0.0 py36_0 conda-forge pyparsing 2.3.1 py_0 conda-forge pyproj 1.9.6 py36h9c430a6_1000 conda-forge pyqt 5.6.0 py36hc26a216_1008 conda-forge pyrsistent 0.14.11 py36h1de35cc_0 conda-forge pyshp 2.1.0 py_0 conda-forge pysocks 1.6.8 py36_1002 conda-forge pytables 3.4.4 py36h0af8b44_1004 conda-forge pytest 4.3.0 py36_0 conda-forge pytest-benchmark 3.0.0 py36_1001 conda-forge python 3.6.7 h4a56312_1002 conda-forge python-dateutil 2.8.0 py_0 conda-forge python-graphviz 0.10.1 py_0 conda-forge python-snappy 0.5.3 py36h8522eec_1000 conda-forge pytz 2018.9 py_0 conda-forge pyviz_comms 0.7.0 py_0 conda-forge pywavelets 1.0.2 py36h917ab60_0 conda-forge pyyaml 3.13 py36h1de35cc_1001 conda-forge pyzmq 18.0.1 py36h4cc6ddd_0 conda-forge qt 5.6.2 h822fa55_1013 conda-forge qtconsole 4.4.3 py_0 conda-forge rasterio 1.0.21 py36h3683dd5_0 conda-forge readline 7.0 hcfe32e1_1001 conda-forge requests 2.21.0 py36_1000 conda-forge rtree 0.8.3 py36_1000 conda-forge scikit-image 0.14.2 py36h0a44026_1 conda-forge scipy 1.2.1 py36_blas_openblash486cb9f_0 [blas_openblas] conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 40.8.0 py36_0 conda-forge shapely 1.6.4 py36h2bcc7ef_1002 conda-forge sip 4.18.1 py36h0a44026_1000 conda-forge six 1.12.0 py36_1000 conda-forge snappy 1.1.7 h0a44026_1002 conda-forge snuggs 1.4.3 py_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge soupsieve 1.8 py36_0 conda-forge sqlite 3.26.0 h1765d9f_1001 conda-forge statsmodels 0.9.0 py36h917ab60_1000 conda-forge streamz 0.2.0 pypi_0 pypi tblib 1.3.2 py_1 conda-forge terminado 0.8.1 py36_1001 conda-forge testpath 0.3.1 pypi_0 pypi thrift 0.11.0 py36h0a44026_1001 conda-forge tk 8.6.9 ha441bb4_1000 conda-forge toolz 0.9.0 py_1 conda-forge tornado 6.0.1 py36h1de35cc_0 conda-forge traitlets 4.3.2 py36_1000 conda-forge tzcode 2018g h1de35cc_1001 conda-forge udunits2 2.2.27.6 h776b7f1_1001 conda-forge urllib3 1.24.1 py36_1000 conda-forge wcwidth 0.1.7 py_1 conda-forge webargs 5.1.2 pypi_0 pypi webencodings 0.5.1 py_1 conda-forge wheel 0.33.1 py36_0 conda-forge widgetsnbextension 3.4.2 py36_1000 conda-forge xarray 0.11.3 py36_0 conda-forge xerces-c 3.2.2 h44e365a_1001 conda-forge xz 5.2.4 h1de35cc_1001 conda-forge yaml 0.1.7 h1de35cc_1001 conda-forge zeromq 4.2.5 h0a44026_1006 conda-forge zict 0.1.4 py_0 conda-forge zlib 1.2.11 h1de35cc_1004 conda-forge ``` Thanks for any help!
ceball commented 5 years ago

I think examples/environment.yml is probably out of date - the dependencies are quite different from the current ones (see https://github.com/pyviz/datashader/blob/master/setup.py#L32, which is the basis for the conda package).

In the short term, maybe you could conda install -c pyviz datashader and then remove --force it (to leave the dependencies), then do pip install -e . --no-deps in the git repo? I have not tested this, and a current datashader developer might have a better idea. Or maybe you could look to see what pins there are in setup.py (e.g. see the pytest and pytest-benchmark pins...), and apply them to your existing environment?

The plan for datashader was to specify the dependencies in one place, and generate both the conda package recipe and examples/environment.yml from that (by filtering/pinning/combining groups as necessary for each one, since they have different purposes). I think we've managed to start doing that in some other pyviz projects (e.g. pyviz itself), but we're not there yet.

jbednar commented 5 years ago

Sounds about right. I think @jsignell can fix environment.yml to be generated from the actual dependencies, so that it matches what we use for testing. But I don't think any removing is needed; I just do conda install -c pyviz datashader then pip install -e . inside the datashader repo, which is normally sufficient to ensure that the repo is used rather than the conda package.

ivirshup commented 5 years ago

Thanks for the advice!

I'm still having trouble though. Here's what I've run so far:

conda create -n datashader
conda activate datashader
conda install --deps-only -yc pyviz datashader

At this point, pytest isn't in the environment, so I've manually added the test requirements from the setup.py:

conda install -y "pytest==3.9.3" "pytest-benchmark==3.0.0" "flake8" "nbsmoke>=0.2.6" "fastparquet>=0.1.6"
pip install -e .

Running pytest at this point throws a tornado error:

Call & traceback ``` (datashader) isaac@mimir ~/github/datashader$ pytest ✭speedup_connectedges Traceback (most recent call last): File "/Users/isaac/miniconda3/envs/datashader/bin/pytest", line 11, in sys.exit(main()) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/_pytest/config/__init__.py", line 56, in main config = _prepareconfig(args, plugins) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/_pytest/config/__init__.py", line 193, in _prepareconfig pluginmanager=pluginmanager, args=args File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__ return self._hookexec(self, self.get_hookimpls(), kwargs) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pluggy/manager.py", line 68, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pluggy/manager.py", line 62, in firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall gen.send(outcome) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/_pytest/helpconfig.py", line 93, in pytest_cmdline_parse config = outcome.get_result() File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(*args) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/_pytest/config/__init__.py", line 663, in pytest_cmdline_parse self.parse(args) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/_pytest/config/__init__.py", line 835, in parse self._preparse(args, addopts=addopts) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/_pytest/config/__init__.py", line 787, in _preparse self.pluginmanager.load_setuptools_entrypoints("pytest11") File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pluggy/manager.py", line 274, in load_setuptools_entrypoints plugin = ep.load() File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load return self.resolve() File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/nbsmoke/__init__.py", line 94, in import nbconvert File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/nbconvert/__init__.py", line 7, in from . import postprocessors File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/nbconvert/postprocessors/__init__.py", line 5, in from .serve import ServePostProcessor File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/nbconvert/postprocessors/serve.py", line 19, in class ProxyHandler(web.RequestHandler): File "/Users/isaac/miniconda3/envs/datashader/lib/python3.7/site-packages/nbconvert/postprocessors/serve.py", line 21, in ProxyHandler @web.asynchronous AttributeError: module 'tornado.web' has no attribute 'asynchronous' ```
Output of `conda list` ``` # packages in environment at /Users/isaac/miniconda3/envs/datashader: # # Name Version Build Channel appnope 0.1.0 py37_0 asn1crypto 0.24.0 py37_0 atomicwrites 1.3.0 py_0 attrs 19.1.0 py_0 backcall 0.1.0 py37_0 beautifulsoup4 4.7.1 py37_1 blas 1.0 mkl bleach 3.1.0 py37_0 bokeh 1.0.4 py37_0 ca-certificates 2019.1.23 0 certifi 2019.3.9 py37_0 cffi 1.12.2 py37hb5b8e2f_1 chardet 3.0.4 py37_1 click 7.0 py37_0 cloudpickle 0.8.0 py37_0 colorcet 1.0.1 py_0 pyviz cryptography 2.6.1 py37ha12b0ac_0 cycler 0.10.0 py37_0 cytoolz 0.9.0.1 py37h1de35cc_1 dask 1.1.4 py_0 dask-core 1.1.4 py_0 datashader 0.7.0a1.post5+g7fa33cb dev_0 datashape 0.5.4 py37_1 decorator 4.3.2 py37_0 distributed 1.26.0 py37_1 entrypoints 0.3 py37_0 fastparquet 0.2.1 py37h1d22016_1 flake8 3.7.7 py37_0 freetype 2.9.1 hb4e5f40_0 heapdict 1.0.0 py37_2 idna 2.8 py37_0 imageio 2.5.0 py37_0 intel-openmp 2019.1 144 ipykernel 5.1.0 py37h39e3cac_0 ipython 7.3.0 py37h39e3cac_0 ipython_genutils 0.2.0 py37_0 jedi 0.13.3 py37_0 jinja2 2.10 py37_0 jpeg 9b he5867d9_2 jsonschema 3.0.1 py37_0 jupyter_client 5.2.4 py37_0 jupyter_core 4.4.0 py37_0 kiwisolver 1.0.1 py37h0a44026_0 libcxx 4.0.1 hcfea43d_1 libcxxabi 4.0.1 hcfea43d_1 libedit 3.1.20181209 hb402a30_0 libffi 3.2.1 h475c297_4 libgfortran 3.0.1 h93005f0_2 libpng 1.6.36 ha441bb4_0 libsodium 1.0.16 h3efe00b_0 libtiff 4.0.10 hcb84e12_2 llvmlite 0.27.0 py37h8c7ce04_0 locket 0.2.0 py37_1 markupsafe 1.1.1 py37h1de35cc_0 matplotlib 3.0.2 py37h54f8f79_0 mccabe 0.6.1 py37_1 mistune 0.8.4 py37h1de35cc_0 mkl 2019.1 144 mkl_fft 1.0.10 py37h5e564d8_0 mkl_random 1.0.2 py37h27c97d8_0 more-itertools 5.0.0 py37_0 msgpack-python 0.6.1 py37h04f5b5a_1 multipledispatch 0.6.0 py37_0 nbconvert 5.3.1 py37_0 nbformat 4.4.0 py37_0 nbsmoke 0.2.7 py_0 conda-forge ncurses 6.1 h0a44026_1 networkx 2.2 py37_1 numba 0.42.0 py37h6440ff4_0 numpy 1.16.2 py37hacdab7b_0 numpy-base 1.16.2 py37h6575580_0 olefile 0.46 py37_0 openssl 1.1.1b h1de35cc_1 packaging 19.0 py37_0 pandas 0.24.1 py37h0a44026_0 pandoc 2.2.3.2 0 pandocfilters 1.4.2 py37_1 param 1.8.2 py_0 pyviz parso 0.3.4 py37_0 partd 0.3.9 py37_0 pexpect 4.6.0 py37_0 pickleshare 0.7.5 py37_0 pillow 5.4.1 py37hb68e598_0 pip 19.0.3 py37_0 pluggy 0.9.0 py37_0 prompt_toolkit 2.0.9 py37_0 psutil 5.6.1 py37h1de35cc_0 ptyprocess 0.6.0 py37_0 py 1.8.0 py37_0 pycodestyle 2.5.0 py37_0 pycparser 2.19 py37_0 pyct 0.4.6 py_0 pyviz pyct-core 0.4.6 py_0 pyviz pyflakes 2.1.1 py37_0 pygments 2.3.1 py37_0 pyopenssl 19.0.0 py37_0 pyparsing 2.3.1 py37_0 pyrsistent 0.14.11 py37h1de35cc_0 pysocks 1.6.8 py37_0 pytest 3.9.3 py37_0 pytest-benchmark 3.0.0 py37_1001 conda-forge python 3.7.2 haf84260_0 python-dateutil 2.8.0 py37_0 pytz 2018.9 py37_0 pywavelets 1.0.2 py37h1d22016_0 pyyaml 3.13 py37h1de35cc_0 pyzmq 18.0.0 py37h0a44026_0 readline 7.0 h1de35cc_5 requests 2.21.0 py37_0 scikit-image 0.14.1 py37h0a44026_0 scipy 1.2.1 py37h1410ff5_0 setuptools 40.8.0 py37_0 six 1.12.0 py37_0 sortedcontainers 2.1.0 py37_0 soupsieve 1.7.1 py37_0 sqlite 3.27.2 ha441bb4_0 tblib 1.3.2 py37_0 testpath 0.3.1 py37_0 thrift 0.11.0 py37h0a44026_0 tk 8.6.8 ha441bb4_0 toolz 0.9.0 py37_0 tornado 6.0.1 py37h1de35cc_0 traitlets 4.3.2 py37_0 urllib3 1.24.1 py37_0 wcwidth 0.1.7 py37_0 webencodings 0.5.1 py37_1 wheel 0.33.1 py37_0 xarray 0.11.3 py37_0 xz 5.2.4 h1de35cc_4 yaml 0.1.7 hc338f04_2 zeromq 4.3.1 h0a44026_3 zict 0.1.4 py37_0 zlib 1.2.11 h1de35cc_3 zstd 1.3.7 h5bba6e5_0 ```
ivirshup commented 5 years ago

Progress! I got rid of the tornado error by downgrading tornado. Now pytest runs, but for some reason it's not finding all the tests. Here's the full set of commands to get to my current environment:

conda create -n datashader
conda activate datashader
conda install --deps-only -yc pyviz datashader
conda install -y "pytest==3.9.3" "pytest-benchmark==3.0.0" "flake8" "nbsmoke>=0.2.6" "fastparquet>=0.1.6"
conda install "tornado<=6"
pip install -e .

However, not all the tests are running. I think a similar issue is happening with the travis builds. In the build for my PR (#725) I noticed travis was passing while appveyor wasn't. Neither should pass, since there are two test which are failing, since I think those tests have a bug. Travis just isn't running those tests (travis logs don't show tests from test_bundling.py).

Looking into this a little further, pytest --collect-only . collects 425 tests (skipping 12) from the conda environment, but using a pip installed pytest (v4.3.0) 461 items are collected with 5 skipped.

Running pytest --collect-only --disable-warnings . | pbcopy I find:

conda env collected tests ``` ============================= test session starts ============================== platform darwin -- Python 3.7.2, pytest-3.9.3, py-1.8.0, pluggy-0.9.0 -- /Users/isaac/miniconda3/envs/datashader/bin/python cachedir: .pytest_cache benchmark: 3.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=5.00us max_time=1.00s calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /Users/isaac/github/datashader, inifile: tox.ini plugins: benchmark-3.0.0, nbsmoke-0.2.7 collecting ... collected 425 items / 12 skipped -Series]'> -]'> ]'> =================== 12 skipped, 14 warnings in 5.56 seconds ==================== ```
pytest 4.3.0 collected tests ``` ============================= test session starts ============================== platform darwin -- Python 3.7.2, pytest-4.3.0, py-1.8.0, pluggy-0.8.1 -- /usr/local/opt/python/bin/python3.7 cachedir: .pytest_cache rootdir: /Users/isaac/github/datashader, inifile: tox.ini plugins: xdist-1.26.0, parallel-0.0.9, mock-1.10.1, helpers-namespace-2019.1.8, forked-1.0.1, cov-2.6.1, pylama-7.6.6 collecting ... collected 461 items / 5 skipped / 456 selected Assert that when two triangles share an edge that would normally get double-drawn, the edge is only drawn for the rightmost (or bottommost) triangle. Assert that when two triangles share an edge that would normally get double-drawn, the edge is only drawn for the rightmost (or bottommost) triangle. Assert that when two triangles share an edge that would normally get double-drawn, the edge is only drawn for the rightmost (or bottommost) triangle. Assert that when two triangles share an edge that would normally get double-drawn, the edge is only drawn for the rightmost (or bottommost) triangle. Assert that when two triangles share an edge that would normally get double-drawn, the edge is only drawn for the rightmost (or bottommost) triangle. Assert that when two triangles share an edge that would normally get double-drawn, the edge is only drawn for the rightmost (or bottommost) triangle. -Series]> -]> ]> Assert slope transfer function Assert aspect transfer function Assert Simple Hillshade transfer function Assert aspect transfer function Assert that we draw triangles properly, without interpolation enabled. Assert that we draw triangles properly, with interpolation enabled. Assert that we draw subpixel triangles properly, both with and without interpolation. Check that point deliberately chosen to be on the upper bound but with a similar-magnitudes subtraction error like that which could occur in extend line does indeed get mapped to last pixel. Assert that when two triangles share an edge that would normally get double-drawn, the edge is only drawn for the rightmost (or bottommost) triangle. Assert that triangles are interpolated when vertex values are provided. Assert that weighting the simplices works as expected. Assert that weighting the vertices works as expected. Assert that CCW windings get converted to CW. Assert that the ``mesh`` argument results in the same rasterization, despite the ``vertices`` and ``simplices`` arguments changing. Assert that the trimesh aggregation API properly handles weights on the simplices. ==================== 5 skipped, 16 warnings in 4.73 seconds ==================== ```
jsignell commented 5 years ago

I did:

conda create -n datashader -c pyviz/label/dev datashader
conda activate datashader
pip install -e .[tests]

I was getting 12 skips from having a broken version of skimage:

>>> import skimage
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jsignell/conda/envs/ds-14/lib/python3.7/site-packages/skimage/__init__.py", line 167, in <module>
    from .util.dtype import (img_as_float32,
  File "/Users/jsignell/conda/envs/ds-14/lib/python3.7/site-packages/skimage/util/__init__.py", line 8, in <module>
    from .arraycrop import crop
  File "/Users/jsignell/conda/envs/ds-14/lib/python3.7/site-packages/skimage/util/arraycrop.py", line 8, in <module>
    from numpy.lib.arraypad import _validate_lengths
ImportError: cannot import name '_validate_lengths' from 'numpy.lib.arraypad' (/Users/jsignell/conda/envs/ds-14/lib/python3.7/site-packages/numpy/lib/arraypad.py)

I upgraded scikit-image to 0.14.2 from conda-forge (conda install -c conda-forge scikit-image=0.14.2) and got 5 skips. Then I got rasterio and am down to 3 skips in collection - but that's when I saw failures crop up:

datashader/tests/test_raster.py::test_raster_aggregate_default FAILED                                                                                                                                              
datashader/tests/test_raster.py::test_raster_aggregate_nearest FAILED                                                                                                                                               
datashader/tests/test_raster.py::test_partial_extent_returns_correct_size FAILED
datashader/tests/test_raster.py::test_raster_single_pixel_range_with_padding FAILED
ceball commented 5 years ago

Just an aside: I see above in a few places using pip install -e . in conda environments. Maybe that's ok now with the latest conda, or if you know what you're doing, but I always avoided it because it installs any dependencies not already present using pip, which can be confusing and/or problematic in a conda environment. When I used conda, I always did pip install -e . --no-deps. pip install -e . --no-deps into a conda environment is still not perfect, because e.g. things like entry points will then be done by setuptools rather than by conda.

(python setup.py develop may be worse still depending on what you want, because missing dependencies will be installed with easy_install, you won't be using wheels, and so on...)

jbednar commented 5 years ago

can be confusing and/or problematic in a conda environment

Right; I always conda-install the released package first, then run pip install --no-deps -e ., which uses conda to install the dependencies then sets up the github master as the actual module in use. Of course, that will go wrong if dependencies are different in master than they are in the latest release, but I don't know of any solution that is any better in that respect.

ianthomas23 commented 2 years ago

The examples/environment.yml file is indeed somewhat out of date. Rather than update it, I am inclined to remove it and change the developer install instructions in the README to use pyctdev as we do in our CI. Then the two setup files remain the single source of dependency information for both pip and conda installs. It will be a little confusing for new developers who are not familiar with pyctdev, but if the instructions are followed precisely we can guarantee that the install will work as we use the same workflow in our CI.

jbednar commented 2 years ago

Does pyctdev do better than the conda + pip -e solution?

ianthomas23 commented 2 years ago

Does pyctdev do better than the conda + pip -e solution?

It is not intrinsically any better, but in practice it is better because it uses the setup files to obtain the list of dependencies and they are always up to date. conda install uses the environment.yml file which is not necessary in sync with the setup files because neither developers nor CI use this approach.

jbednar commented 2 years ago

Sorry. I didn't mean conda+environment.yml + pip -e, I meant just conda install -c pyviz datashader to get the latest released datashader, followed by pip install --no-deps -e . in the git repo clone. That's what I always do, and it's been reliable so far despite @ceball's fearmongering above. :-) I do think we should delete the outdated environment.yml file, since there's no automatic mechanism to keep it up to date, and I don't want to start a project to make it be auto-generated to be accurate.

It's true that my method won't always install the latest dependencies, if those have been updated in git master but not yet released, but I haven't noticed that to be a problem in practice. We rarely add new dependencies, and when we do they are typically for specific examples, not the core library itself. We can of course tell people to use pyctdev like the CI does to avoid that problem, but my question is whether using pyctdev provides benefits that are worth the extra cognitive load of learning about it. I'm not sure it does, but am open to learning about some benefit it provides for a developer that is worthwhile.

ianthomas23 commented 2 years ago

I understand now!

I think your approach is absolutely fine now. My default approach is even simpler, just a pip install -e .[tests,examples] in a clean virtual environment, which is fine on Linux but may be less so on other platforms.

OK, so plan A is to remove the environment.yml file and not mention pyctdev any more than we already do. I'll write a PR to do this and update the README with the preferred dev installation process, but before that I will check exactly what does work on the 3 major platforms.