search

pangeo-data / pangeo-stacks

Curated Docker images for use with Jupyter and Pangeo
https://pangeo-data.github.io/pangeo-stacks/
BSD 3-Clause "New" or "Revised" License
17 stars 20 forks source link

Dask lab extension unable to launch new clusters (2019.05.19) #48

Closed scottyhq closed 5 years ago

scottyhq commented 5 years ago

The most recent images did away with pinning most versions: https://github.com/pangeo-data/pangeo-stacks/issues/42

Unfortunately in running these new images, the dask labextension is no longer able to launch new KubeClusters (you can select the latest image to run on this hub: https://nasa.pangeo.io)

Seeing messages such as these:

Failed to load resource: the server responded with a status of 500 ()
clusters.js:146 Uncaught (in promise) Error: Failed to start Dask cluster
    at DaskClusterManager.<anonymous> (clusters.js:146)
    at Generator.next (<anonymous>)
    at fulfilled (clusters.js:4)
serverconnection.js:192 PUT https://nasa.pangeo.io/user/scottyhq/dask/clusters?1558478492967 500
handleRequest @ serverconnection.js:192
makeRequest @ serverconnection.js:75
(anonymous) @ clusters.js:144
(anonymous) @ clusters.js:7
push.eY2S.__awaiter @ clusters.js:3
_launchCluster @ clusters.js:143
onClick @ clusters.js:66
handleMouseDown @ toolbar.js:332
callCallback @ react-dom.development.js:100
invokeGuardedCallbackDev @ react-dom.development.js:138
invokeGuardedCallback @ react-dom.development.js:187
invokeGuardedCallbackAndCatchFirstError @ react-dom.development.js:201
executeDispatch @ react-dom.development.js:461
executeDispatchesInOrder @ react-dom.development.js:483
executeDispatchesAndRelease @ react-dom.development.js:581
executeDispatchesAndReleaseTopLevel @ react-dom.development.js:592
forEachAccumulated @ react-dom.development.js:562
runEventsInBatch @ react-dom.development.js:723
runExtractedEventsInBatch @ react-dom.development.js:732
handleTopLevel @ react-dom.development.js:4477
batchedUpdates$1 @ react-dom.development.js:16660
batchedUpdates @ react-dom.development.js:2131
dispatchEvent @ react-dom.development.js:4556
interactiveUpdates$1 @ react-dom.development.js:16715
interactiveUpdates @ react-dom.development.js:2150
dispatchInteractiveEvent @ react-dom.development.js:4533
clusters.js:146 Uncaught (in promise) Error: Failed to start Dask cluster
    at DaskClusterManager.<anonymous> (clusters.js:146)
    at Generator.next (<anonymous>)
    at fulfilled (clusters.js:4)

And here is a copy of the full conda environment installed:

# packages in environment at /srv/conda/envs/notebook:
#
# Name                    Version                   Build  Channel
adal                      1.2.1                      py_0    conda-forge
alembic                   1.0.9                    pypi_0    pypi
asn1crypto                0.24.0                py36_1003    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     19.1.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
blinker                   1.4                        py_1    conda-forge
bokeh                     1.1.0                    py36_0    conda-forge
ca-certificates           2019.3.9             hecc5488_0    conda-forge
cachetools                2.1.0                      py_0    conda-forge
certifi                   2019.3.9                 py36_0    conda-forge
certipy                   0.1.3                      py_0    conda-forge
cffi                      1.12.3           py36h8022711_0    conda-forge
chardet                   3.0.4                    pypi_0    pypi
click                     7.0                        py_0    conda-forge
cloudpickle               1.0.0                      py_0    conda-forge
configurable-http-proxy   1.3.0                         0    conda-forge
cryptography              2.6.1            py36h72c5cf5_0    conda-forge
cytoolz                   0.9.0.1         py36h14c3975_1001    conda-forge
dask-core                 1.2.2                      py_0    conda-forge
dask-kubernetes           0.8.0                      py_0    conda-forge
dask-labextension         0.3.3                    pypi_0    pypi
dbus                      1.13.6               he372182_0    conda-forge
decorator                 4.4.0                      py_0    conda-forge
defusedxml                0.5.0                      py_1    conda-forge
distributed               1.28.1                   py36_0    conda-forge
entrypoints               0.3                   py36_1000    conda-forge
expat                     2.2.5             hf484d3e_1002    conda-forge
fontconfig                2.13.1            he4413a7_1000    conda-forge
freetype                  2.10.0               he983fc9_0    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
glib                      2.58.3            hf63aee3_1001    conda-forge
google-auth               1.6.3                      py_0    conda-forge
gst-plugins-base          1.14.4            hdf3bae2_1001    conda-forge
gstreamer                 1.14.4            h66beb1c_1001    conda-forge
heapdict                  1.0.0                 py36_1000    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
idna                      2.8                      pypi_0    pypi
ipykernel                 5.1.0           py36h24bf2e0_1002    conda-forge
ipython                   7.5.0            py36h24bf2e0_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.4.2                      py_0    conda-forge
jedi                      0.13.3                   py36_0    conda-forge
jinja2                    2.10.1                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
jsonschema                3.0.1                    py36_0    conda-forge
jupyter                   1.0.0                      py_2    conda-forge
jupyter_client            5.2.4                      py_3    conda-forge
jupyter_console           6.0.0                      py_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterhub                0.9.4                    pypi_0    pypi
jupyterlab                0.35.6                   py36_0    conda-forge
jupyterlab_server         0.2.0                      py_0    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
libblas                   3.8.0               10_openblas    conda-forge
libcblas                  3.8.0               10_openblas    conda-forge
libcurl                   7.64.1               hda55be3_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 8.2.0                hdf63c60_1    defaults
libgfortran-ng            7.3.0                hdf63c60_0    defaults
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0               10_openblas    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libsodium                 1.0.16            h14c3975_1001    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1    defaults
libtiff                   4.0.10            h648cc4a_1001    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.9                h13577e0_0    conda-forge
mako                      1.0.9                    pypi_0    pypi
markupsafe                1.1.1            py36h14c3975_0    conda-forge
mistune                   0.8.4           py36h14c3975_1000    conda-forge
msgpack-python            0.6.1            py36h6bb024c_0    conda-forge
nbconvert                 5.4.1                      py_2    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
nbserverproxy             0.8.8                   py_1000    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
nodejs                    11.14.0              he1b5a44_1    conda-forge
notebook                  5.7.8                    py36_0    conda-forge
nteract-on-jupyter        2.0.12                   pypi_0    pypi
numpy                     1.16.3           py36he5ce36f_0    conda-forge
oauthlib                  3.0.1                      py_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openblas                  0.3.6                h6e990d7_2    conda-forge
openssl                   1.1.1b               h14c3975_1    conda-forge
packaging                 19.0                       py_0    conda-forge
pamela                    1.0.0                      py_0    conda-forge
pandoc                    2.7.2                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.4.0                      py_0    conda-forge
pcre                      8.41              hf484d3e_1003    conda-forge
pexpect                   4.7.0                    py36_0    conda-forge
pickleshare               0.7.5                 py36_1000    conda-forge
pillow                    6.0.0            py36he7afcd5_0    conda-forge
pip                       19.1                     py36_0    conda-forge
prometheus_client         0.6.0                      py_0    conda-forge
prompt_toolkit            2.0.9                      py_0    conda-forge
psutil                    5.6.2            py36h516909a_0    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pyasn1                    0.4.4                      py_1    conda-forge
pyasn1-modules            0.2.4                      py_0    conda-forge
pycparser                 2.19                     py36_1    conda-forge
pycurl                    7.43.0.2         py36h16ce93b_0    conda-forge
pygments                  2.3.1                      py_0    conda-forge
pyjwt                     1.7.1                      py_0    conda-forge
pyopenssl                 19.0.0                   py36_0    conda-forge
pyparsing                 2.4.0                      py_0    conda-forge
pyqt                      5.9.2            py36hcca6a23_0    conda-forge
pyrsistent                0.15.1           py36h516909a_0    conda-forge
pysocks                   1.7.0                    py36_0    conda-forge
python                    3.6.7             h381d211_1004    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
python-editor             1.0.4                    pypi_0    pypi
python-graphviz           0.10.1                   pypi_0    pypi
python-kubernetes         9.0.0                    py36_0    conda-forge
python-oauth2             1.1.0                    pypi_0    pypi
pyyaml                    5.1              py36h14c3975_0    conda-forge
pyzmq                     18.0.1           py36hc4ba49a_1    conda-forge
qt                        5.9.7                h52cfd70_1    conda-forge
qtconsole                 4.4.4                      py_0    conda-forge
readline                  7.0               hf8c457e_1001    conda-forge
requests                  2.21.0                   pypi_0    pypi
requests-oauthlib         1.2.0                      py_0    conda-forge
rsa                       3.4.2                      py_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                41.0.1                   py36_0    conda-forge
sip                       4.19.8          py36hf484d3e_1000    conda-forge
six                       1.12.0                py36_1000    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
sqlalchemy                1.3.3            py36h516909a_0    conda-forge
sqlite                    3.26.0            h67949de_1001    conda-forge
tblib                     1.3.2                      py_1    conda-forge
terminado                 0.8.2                    py36_0    conda-forge
testpath                  0.4.2                   py_1001    conda-forge
tk                        8.6.9             h84994c4_1001    conda-forge
toolz                     0.9.0                      py_1    conda-forge
tornado                   5.1.1           py36h14c3975_1000    conda-forge
traitlets                 4.3.2                 py36_1000    conda-forge
urllib3                   1.24.2                   pypi_0    pypi
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          0.56.0                   py36_0    conda-forge
wheel                     0.33.1                   py36_0    conda-forge
widgetsnbextension        3.4.2                 py36_1000    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.1.7             h14c3975_1001    conda-forge
zeromq                    4.3.1             hf484d3e_1000    conda-forge
zict                      0.1.4                      py_0    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge

JupyterLab v0.35.6
Known labextensions:
   app dir: /srv/conda/envs/notebook/share/jupyter/lab
        @jupyter-widgets/jupyterlab-manager v0.38.1  enabled  OK
        @jupyterlab/hub-extension v0.12.0  enabled  OK
        @pyviz/jupyterlab_pyviz v0.7.2  enabled  OK
        dask-labextension v0.3.0  enabled  OK
        jupyter-leaflet v0.10.2  enabled  OK

pinging @ian-r-rose @jhamman

ian-r-rose commented 5 years ago

@scottyhq Do you have access to the server logs? Do they have any interesting information in them?

I'd still be wary about tornado 6...

scottyhq commented 5 years ago

we've pinned to tornado 5.1.1. I see the following message in the log: [W 2019-05-21 23:04:04.105 SingleUserLabApp handlers:620] object KubeCluster can't be used in 'await' expression

ian-r-rose commented 5 years ago

Hmm, I'm not sure. @mrocklin does this look familiar? I wonder if it's a regression in dask-kubernetes.

scottyhq commented 5 years ago

Actually... i'm suspicious this is something to do with the a new version of repo2docker and some mixing of environments (https://github.com/pangeo-data/pangeo-stacks/issues/47). Because if I list the packages in the 'base' environment we have the following (including tornado 6.0.2):

(base) jovyan@~:conda list
# packages in environment at /srv/conda:
#
# Name                    Version                   Build  Channel
absl-py                   0.7.1                    py36_0    conda-forge
affine                    2.2.2                      py_0    conda-forge
antlr-python-runtime      4.7.2                 py36_1000    conda-forge
appdirs                   1.4.3                      py_1    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
asn1crypto                0.24.0                py36_1003    conda-forge
astor                     0.7.1                      py_0    conda-forge
attrdict                  2.0.1                      py_0    conda-forge
attrs                     19.1.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
beautifulsoup4            4.7.1                 py36_1001    conda-forge
bleach                    3.1.0                      py_0    conda-forge
blinker                   1.4                        py_1    conda-forge
blosc                     1.16.3               hf484d3e_0    conda-forge
bokeh                     1.1.0                    py36_0    conda-forge
boost-cpp                 1.68.0            h11c811c_1000    conda-forge
boto3                     1.9.151                    py_0    conda-forge
botocore                  1.12.151                   py_0    conda-forge
branca                    0.3.1                      py_0    conda-forge
bzip2                     1.0.6             h14c3975_1002    conda-forge
c-ares                    1.15.0            h14c3975_1001    conda-forge
ca-certificates           2019.3.9             hecc5488_0    conda-forge
cachetools                2.1.0                      py_0    conda-forge
cairo                     1.16.0            ha4e643d_1000    conda-forge
cartopy                   0.17.0          py36h0aa2c8f_1004    conda-forge
certifi                   2019.3.9                 py36_0    conda-forge
cf-units                  2.1.1           py36h3010b51_1000    conda-forge
cffi                      1.12.3           py36h8022711_0    conda-forge
cftime                    1.0.3.4         py36hd352d35_1001    conda-forge
chardet                   3.0.4                 py36_1003    conda-forge
click                     7.0                        py_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.5.0                      py_0    conda-forge
climlab                   0.7.3            py36h4c70da7_0    conda-forge
cloudpickle               1.0.0                      py_0    conda-forge
colorcet                  2.0.1                      py_0    conda-forge
conda                     4.6.14                   py36_0    conda-forge
configobj                 5.0.6                      py_0    conda-forge
cryptography              2.6.1            py36h72c5cf5_0    conda-forge
curl                      7.64.1               hf8cf82a_0    conda-forge
cycler                    0.10.0                     py_1    conda-forge
cytoolz                   0.9.0.1         py36h14c3975_1001    conda-forge
dask                      1.2.2                      py_0    conda-forge
dask-core                 1.2.2                      py_0    conda-forge
dask-glm                  0.1.0                         0    conda-forge
dask-ml                   0.12.0                     py_0    conda-forge
datashader                0.7.0                      py_0    conda-forge
datashape                 0.5.4                      py_1    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
decorator                 4.4.0                      py_0    conda-forge
defusedxml                0.5.0                      py_1    conda-forge
descartes                 1.1.0                      py_3    conda-forge
distributed               1.28.1                   py36_0    conda-forge
docopt                    0.6.2                      py_1    conda-forge
docrep                    0.2.5                      py_0    conda-forge
docutils                  0.14                  py36_1001    conda-forge
entrypoints               0.3                   py36_1000    conda-forge
eofs                      1.4.0                      py_0    conda-forge
esmf                      7.1.0             hdfb41a0_1004    conda-forge
esmpy                     7.1.0            py36h24bf2e0_3    conda-forge
expat                     2.2.5             hf484d3e_1002    conda-forge
fasteners                 0.14.1                     py_3    conda-forge
fiona                     1.8.6            py36hf242f0b_3    conda-forge
fontconfig                2.13.1            he4413a7_1000    conda-forge
freetype                  2.10.0               he983fc9_0    conda-forge
freexl                    1.0.5             h14c3975_1002    conda-forge
future                    0.17.1                py36_1000    conda-forge
gast                      0.2.2                      py_0    conda-forge
gcsfs                     0.2.2                      py_0    conda-forge
gdal                      2.4.1            py36hf242f0b_0    conda-forge
geopandas                 0.5.0                      py_2    conda-forge
geos                      3.7.1             hf484d3e_1000    conda-forge
geotiff                   1.4.3             h1105359_1000    conda-forge
geoviews                  1.6.2                      py_0    conda-forge
geoviews-core             1.6.2                      py_0    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
giflib                    5.1.7                h516909a_1    conda-forge
glib                      2.58.3            hf63aee3_1001    conda-forge
google-auth               1.6.3                      py_0    conda-forge
google-auth-oauthlib      0.3.0                      py_0    conda-forge
grpcio                    1.16.1           py36hf8bcb03_1    defaults
gst-plugins-base          1.14.4            hdf3bae2_1001    conda-forge
gstreamer                 1.14.4            h66beb1c_1001    conda-forge
gsw                       3.3.1            py36h516909a_0    conda-forge
h5netcdf                  0.7.1                      py_0    conda-forge
h5py                      2.9.0           nompi_py36hf008753_1102    conda-forge
hdf4                      4.2.13            h9a582f1_1002    conda-forge
hdf5                      1.10.4          nompi_h3c11f04_1106    conda-forge
heapdict                  1.0.0                 py36_1000    conda-forge
holoviews                 1.12.2                     py_0    conda-forge
hvplot                    0.4.0                      py_1    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
idna                      2.8                   py36_1000    conda-forge
imageio                   2.5.0                    py36_0    conda-forge
intake                    0.5.0                      py_0    conda-forge
intake-esm                2019.5.11                  py_0    conda-forge
intake-xarray             0.3.0                      py_0    conda-forge
ipykernel                 5.1.1            py36h24bf2e0_0    conda-forge
ipyleaflet                0.10.2                   py36_0    conda-forge
ipython                   7.5.0            py36h24bf2e0_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.4.2                      py_0    conda-forge
iris                      2.2.0                 py36_1003    conda-forge
jedi                      0.13.3                   py36_0    conda-forge
jinja2                    2.10.1                     py_0    conda-forge
jmespath                  0.9.4                      py_0    conda-forge
joblib                    0.13.2                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
json-c                    0.13.1            h14c3975_1001    conda-forge
jsonschema                3.0.1                    py36_0    conda-forge
jupyter                   1.0.0                      py_2    conda-forge
jupyter_client            5.2.4                      py_3    conda-forge
jupyter_console           6.0.0                      py_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
kealib                    1.4.10            h1978553_1003    conda-forge
keras                     2.2.4                    py36_1    conda-forge
keras-applications        1.0.7                      py_1    conda-forge
keras-preprocessing       1.0.9                      py_1    conda-forge
kiwisolver                1.1.0            py36hc9558a2_0    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
libblas                   3.8.0               10_openblas    conda-forge
libcblas                  3.8.0               10_openblas    conda-forge
libcurl                   7.64.1               hda55be3_0    conda-forge
libdap4                   3.19.1            hd48c02d_1000    conda-forge
libedit                   3.1.20181209         hc058e9b_0    defaults
libffi                    3.2.1                hd88cf55_4    defaults
libgcc-ng                 8.2.0                hdf63c60_1    defaults
libgdal                   2.4.1                hdb8f723_0    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0    defaults
libgpuarray               0.7.6             h14c3975_1003    conda-forge
libiconv                  1.15              h516909a_1005    conda-forge
libkml                    1.3.0             h328b03d_1009    conda-forge
liblapack                 3.8.0               10_openblas    conda-forge
libnetcdf                 4.6.2             hbdf4f91_1001    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libpq                     11.3                 h4e4e079_0    conda-forge
libprotobuf               3.7.1                h8b12597_0    conda-forge
libsodium                 1.0.16            h14c3975_1001    conda-forge
libspatialindex           1.9.0                he1b5a44_0    conda-forge
libspatialite             4.3.0a            hb5ec416_1026    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1    defaults
libtiff                   4.0.10            h648cc4a_1001    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.9                h13577e0_0    conda-forge
libxslt                   1.1.32            h4785a14_1002    conda-forge
llvmlite                  0.28.0           py36hdbcaa40_0    conda-forge
locket                    0.2.0                      py_2    conda-forge
lxml                      4.3.3            py36h7ec2d77_0    conda-forge
lz4                       2.1.6           py36hd79334b_1001    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
mako                      1.0.7                      py_1    conda-forge
mapclassify               2.0.1                      py_0    conda-forge
markdown                  2.6.11                     py_0    conda-forge
markupsafe                1.1.1            py36h14c3975_0    conda-forge
matplotlib                3.0.3                    py36_1    conda-forge
matplotlib-base           3.0.3            py36h5f35d83_1    conda-forge
mechanicalsoup            0.11.0                     py_0    conda-forge
mercantile                1.0.4                    pypi_0    pypi
metpy                     0.10.0                py36_1001    conda-forge
mistune                   0.8.4           py36h14c3975_1000    conda-forge
mock                      3.0.5                    py36_0    conda-forge
monotonic                 1.5                        py_0    conda-forge
mpi                       1.0                       mpich    conda-forge
mpich                     3.2.1             h1c2f66e_1008    conda-forge
msgpack-numpy             0.4.4.3                    py_0    conda-forge
msgpack-python            0.6.1            py36h6bb024c_0    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
munch                     2.3.2                      py_0    conda-forge
nbconvert                 5.5.0                      py_0    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1                  he6710b0_1    defaults
netcdf-fortran            4.4.5             hea25ff8_1000    conda-forge
netcdf4                   1.5.1.2          py36had58050_0    conda-forge
networkx                  2.3                        py_0    conda-forge
ninja                     1.9.0                h6bb024c_0    conda-forge
notebook                  5.7.8                    py36_0    conda-forge
numba                     0.43.1           py36hf2d7682_0    conda-forge
numcodecs                 0.6.3            py36hf484d3e_0    conda-forge
numpy                     1.16.3           py36he5ce36f_0    conda-forge
oauthlib                  3.0.1                      py_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openblas                  0.3.6                h6e990d7_2    conda-forge
openjpeg                  2.3.1                h58a6597_0    conda-forge
openssl                   1.1.1b               h14c3975_1    conda-forge
owslib                    0.17.1                     py_0    conda-forge
packaging                 19.0                       py_0    conda-forge
pandas                    0.24.2           py36hf484d3e_0    conda-forge
pandoc                    2.7.2                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
panel                     0.5.1                h24bf2e0_2    conda-forge
param                     1.9.0                      py_0    conda-forge
parso                     0.4.0                      py_0    conda-forge
partd                     0.3.9                      py_0    conda-forge
pcre                      8.41              hf484d3e_1003    conda-forge
pexpect                   4.7.0                    py36_0    conda-forge
pickleshare               0.7.5                 py36_1000    conda-forge
pillow                    6.0.0            py36he7afcd5_0    conda-forge
pint                      0.9                      py36_2    conda-forge
pip                       19.1                     py36_0    conda-forge
pixman                    0.34.0            h14c3975_1003    conda-forge
pooch                     0.2.1                 py36_1000    conda-forge
poppler                   0.67.0               h2ad7f00_6    conda-forge
poppler-data              0.4.9                         1    conda-forge
postgresql                11.3                 hbeee2d4_0    conda-forge
proj4                     5.2.0             h14c3975_1001    conda-forge
prometheus_client         0.6.0                      py_0    conda-forge
prompt_toolkit            2.0.9                      py_0    conda-forge
protobuf                  3.7.1            py36he1b5a44_0    conda-forge
psutil                    5.6.2            py36h516909a_0    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pyasn1                    0.4.4                      py_1    conda-forge
pyasn1-modules            0.2.4                      py_0    conda-forge
pycosat                   0.6.3           py36h14c3975_1001    conda-forge
pycparser                 2.19                     py36_1    conda-forge
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pydap                     3.2.2                 py36_1000    conda-forge
pyepsg                    0.4.0                      py_0    conda-forge
pygments                  2.4.0                      py_0    conda-forge
pygpu                     0.7.6           py36h3010b51_1000    conda-forge
pyjwt                     1.7.1                      py_0    conda-forge
pykdtree                  1.3.1           py36h3010b51_1002    conda-forge
pyke                      1.1.1                 py36_1000    conda-forge
pyopenssl                 19.0.0                   py36_0    conda-forge
pyorbital                 1.5.0                      py_0    conda-forge
pyparsing                 2.4.0                      py_0    conda-forge
pyproj                    1.9.6           py36hc0953d3_1000    conda-forge
pyqt                      5.9.2            py36hcca6a23_0    conda-forge
pyresample                1.12.3           py36hf2d7682_0    conda-forge
pyrsistent                0.15.2           py36h516909a_0    conda-forge
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.0                    py36_0    conda-forge
pyspectral                0.8.8                      py_0    conda-forge
pyspharm                  1.0.9           py36h6091dcd_1003    conda-forge
python                    3.6.7             h381d211_1004    conda-forge
python-blosc              1.8.1            py36hf484d3e_0    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
python-geotiepoints       1.1.8            py36hd352d35_0    conda-forge
python-snappy             0.5.4            py36hee44bf9_0    conda-forge
pytorch-cpu               1.0.1            py36hf484d3e_0    conda-forge
pytz                      2019.1                     py_0    conda-forge
pyviz_comms               0.7.2                      py_0    conda-forge
pywavelets                1.0.3            py36hd352d35_1    conda-forge
pyyaml                    5.1              py36h14c3975_0    conda-forge
pyzmq                     18.0.1           py36hc4ba49a_1    conda-forge
qt                        5.9.7                h52cfd70_1    conda-forge
qtconsole                 4.4.4                      py_0    conda-forge
rasterio                  1.0.23           py36h5b3f9e8_0    conda-forge
readline                  7.0                  h7b6447c_5    defaults
regionmask                0.4.0                    pypi_0    pypi
requests                  2.22.0                   py36_0    conda-forge
requests-oauthlib         1.2.0                      py_0    conda-forge
rio-cogeo                 1.0.0                    pypi_0    pypi
rsa                       3.4.2                      py_1    conda-forge
rtree                     0.8.3           py36h666c49c_1002    conda-forge
ruamel                    1.0                      py36_0    conda-forge
ruamel.yaml               0.15.96          py36h516909a_0    conda-forge
ruamel_yaml               0.15.71         py36h14c3975_1000    conda-forge
s3fs                      0.2.1                      py_0    conda-forge
s3transfer                0.2.0                    py36_0    conda-forge
satpy                     0.15.1             pyh326bf55_0    conda-forge
scikit-image              0.15.0           py36he1b5a44_0    conda-forge
scikit-learn              0.21.1           py36h627018c_0    conda-forge
scipy                     1.2.1            py36h09a28d5_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                41.0.1                   py36_0    conda-forge
shapely                   1.6.4           py36h2afed24_1004    conda-forge
sip                       4.19.8          py36hf484d3e_1000    conda-forge
six                       1.12.0                py36_1000    conda-forge
snappy                    1.1.7             hf484d3e_1002    conda-forge
snuggs                    1.4.6                      py_0    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
soupsieve                 1.9.1                    py36_0    conda-forge
sqlite                    3.27.2               h7b6447c_0    defaults
supermercado              0.0.5                    pypi_0    pypi
tblib                     1.3.2                      py_1    conda-forge
tensorboard               1.13.1                   py36_0    conda-forge
tensorflow                1.13.1                   py36_0    conda-forge
tensorflow-estimator      1.13.0                     py_0    defaults
termcolor                 1.1.0                      py_2    conda-forge
terminado                 0.8.2                    py36_0    conda-forge
testpath                  0.3.1                    py36_1    conda-forge
theano                    1.0.3                    py36_0    conda-forge
tk                        8.6.9             h84994c4_1001    conda-forge
toolz                     0.9.0                      py_1    conda-forge
tornado                   6.0.2            py36h516909a_0    conda-forge
tqdm                      4.32.1                     py_0    conda-forge
traitlets                 4.3.2                 py36_1000    conda-forge
traittypes                0.2.1                      py_1    conda-forge
trollimage                1.8.0                      py_0    conda-forge
trollsift                 0.3.2                      py_0    conda-forge
tzcode                    2018g             h14c3975_1001    conda-forge
udunits2                  2.2.27.6          h4e0c4b3_1001    conda-forge
urllib3                   1.24.3                   py36_0    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
webob                     1.8.5                      py_0    conda-forge
werkzeug                  0.15.4                     py_0    conda-forge
wheel                     0.33.4                   py36_0    conda-forge
widgetsnbextension        3.4.2                 py36_1000    conda-forge
windspharm                1.7.0                 py36_1000    conda-forge
xarray                    0.12.1                     py_0    conda-forge
xerces-c                  3.2.2             hac72e42_1001    conda-forge
xesmf                     0.1.1                    py36_1    conda-forge
xgcm                      0.2.0                      py_0    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.9             h516909a_1004    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.7             h14c3975_1000    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xrft                      0.2.0                      py_0    conda-forge
xz                        5.2.4                h14c3975_4    defaults
yaml                      0.1.7                had09818_2    defaults
zarr                      2.3.1                    py36_0    conda-forge
zeromq                    4.3.1             hf484d3e_1000    conda-forge
zict                      0.1.4                      py_0    conda-forge
zlib                      1.2.11               h7b6447c_3    defaults
mrocklin commented 5 years ago

My first guess would be JupyterLab + Tornado 6 conflicts. I don't think that dask-kubernetes has changed a ton recently, but @jhamman might know more.

ian-r-rose commented 5 years ago

dask-kubernetes had a release four days ago. Worth checking, I think.

mrocklin commented 5 years ago

Fair point

scottyhq commented 5 years ago

@ian-r-rose - given multiple conda environments on a jupyterhub, which does dask labextension use by default?

ian-r-rose commented 5 years ago

Whichever one is used to launch JupyterLab, I think.

jhamman commented 5 years ago

@scottyhq - can you try with a LocalCluster and see if that works? That will help determine if its in KubeCluster or not.

scottyhq commented 5 years ago

Just to clarify, launching a KubeCluster programmatically works, and I can use the 'search' glass to find it and activate all buttons. It is the 'clusters +new' part that is non-responsive. If anyone wants to enter the hub and explore further, see the hub link in the first comment.

import dask
from dask_kubernetes import KubeCluster
from dask.distributed import Client
from dask.distributed import wait, progress
cluster = KubeCluster(n_workers=2)
cluster
mrocklin commented 5 years ago

Oh, yeah, launching clusters through the lab extension requires that they support async/await. This is the whole reason for the async development branch in dask kubernetes.

On Tue, May 21, 2019, 6:42 PM Scott Henderson notifications@github.com wrote:

Just to clarify, launching a KubeCluster programmatically works, and I can use the 'search' glass to find it and activate all buttons. It is the 'clusters +new' part that is non-responsive. If anyone wants to enter the hub and explore further, see the hub link in the first comment.

import daskfrom dask_kubernetes import KubeClusterfrom dask.distributed import Clientfrom dask.distributed import wait, progress cluster = KubeCluster(n_workers=2) cluster

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo-stacks/issues/48?email_source=notifications&email_token=AACKZTD2ZY5RFH73PXD76UDPWSCEZA5CNFSM4HOPJQI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV5PUXI#issuecomment-494598749, or mute the thread https://github.com/notifications/unsubscribe-auth/AACKZTCJQUTRWNF33G65MXTPWSCEZANCNFSM4HOPJQIQ .

ian-r-rose commented 5 years ago

Has that not been merged/published yet?

jhamman commented 5 years ago

Has that not been merged/published yet?

No, its still sitting in the dev brach.

Binder

This binder does a few things:

Full specs here: https://github.com/jhamman/pangeo_test_binder/tree/dask_dev

ian-r-rose commented 5 years ago

Is there anything that should be done to move forwards with an async-aware dask-kubernetes? I hadn't realized we were still blocking on that.

mrocklin commented 5 years ago

There are a couple things to do, yes. Mostly it needs to be used and bugs need to be found and fixed. I plan to write up the state of things and a few possible plans as an issue later this week.

TomAugspurger commented 5 years ago

@mrocklin I should have time to push on async KubeCluster (not this week, but probably next week). Is that something you'd like me to take on?

mrocklin commented 5 years ago

That would be very welcome. @quasiben has some work. Also, I can also imagine a future in which we make a Worker-like class that manages a Pod, and then just use SpecCluster around that class. I'd be happy to have a chat any time about options here if you're interested.

On Tue, Jun 4, 2019 at 1:56 PM Tom Augspurger notifications@github.com wrote:

@mrocklin https://github.com/mrocklin I should have time to push on async KubeCluster (not this week, but probably next week). Is that something you'd like me to take on?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo-stacks/issues/48?email_source=notifications&email_token=AACKZTAV4PXNWBSIPL3OUBDPY3JJTA5CNFSM4HOPJQI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW53IIQ#issuecomment-498840610, or mute the thread https://github.com/notifications/unsubscribe-auth/AACKZTBTX3WSF6SSCHKYOP3PY3JJTANCNFSM4HOPJQIQ .

ian-r-rose commented 5 years ago

If it will take a significant amount of time, we can also back out the changes that make dask-labextension expect an asynchronously started cluster.

quasiben commented 5 years ago

@TomAugspurger there are two PRs for async kube if you are interested:

I would suggest looking at @mrocklin 's before mine.

scottyhq commented 5 years ago

noting that once upcoming async changes to dask-kubernetes are merged and released we should bump to dask-labextension > 1.0

https://github.com/dask/dask-kubernetes/pull/162