Feedback from a young graduate in CS & Applied Maths

[jjerphan]

Hi Coiled!

I started experimenting exploring the Dashboard and the Python API. I have been playing 20 minutes with some Clusters and reading the documentation for about 30 minutes.

As a Computer Scientist/Applied Mathematician familiar with Python, the python ecosystem and Dask everything ran smoothly, was clear and modular.

I just would like to report a few things that can be easily fixed:

• python 3.9 doesn't seem to be supported
• a few indications on the Dashboard can be improved.

Python 3.9

I am using python 3.9 (distributed from Fedora's repository) when I can. Looking at Pypi, Coiled is to be usable for this version.

I've managed to install it within a virtual environment (manually with the venv module) without any problems. Though, when running the preliminary snippet, I get an error when spawning a coiled.Cluster:

Python 3.9.1 (default, Dec  8 2020, 00:00:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import coiled

In [2]: cluster = coiled.Cluster(n_workers=10)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-4a061294e4f0> in <module>
----> 1 cluster = coiled.Cluster(n_workers=10)

~/.virtualenvs/coiled/lib64/python3.9/site-packages/coiled/cluster.py in __init__(self, n_workers, configuration, software, worker_cpu, worker_gpu, worker_memory, worker_class, worker_options, scheduler_cpu, scheduler_memory, scheduler_class, scheduler_options, name, asynchronous, cloud, account, shutdown_on_close, backend_options, credentials)
    151         self._name = "coiled.Cluster"  # Used in Dask's Cluster._ipython_display_
    152         if not self.asynchronous:
--> 153             self.sync(self._start)
    154 
    155     @property

~/.virtualenvs/coiled/lib64/python3.9/site-packages/distributed/deploy/cluster.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    187             return future
    188         else:
--> 189             return sync(self.loop, func, *args, **kwargs)
    190 
    191     def _log(self, log):

~/.virtualenvs/coiled/lib64/python3.9/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    338     if error[0]:
    339         typ, exc, tb = error[0]
--> 340         raise exc.with_traceback(tb)
    341     else:
    342         return result[0]

~/.virtualenvs/coiled/lib64/python3.9/site-packages/distributed/utils.py in f()
    322             if callback_timeout is not None:
    323                 future = asyncio.wait_for(future, callback_timeout)
--> 324             result[0] = yield future
    325         except Exception as exc:
    326             error[0] = sys.exc_info()

~/.virtualenvs/coiled/lib64/python3.9/site-packages/tornado/gen.py in run(self)
    760 
    761                     try:
--> 762                         value = future.result()
    763                     except Exception:
    764                         exc_info = sys.exc_info()

~/.virtualenvs/coiled/lib64/python3.9/site-packages/coiled/cluster.py in _start(self)
    190                         f"  coiled.create_cluster_configuration(name='{self.configuration}', software='{self.configuration}')"  # noqa: E501
    191                     )
--> 192                 raise ValueError(error_msg)
    193 
    194             self.cluster_id = await self.cloud.create_cluster(

ValueError: Cluster configuration 'coiled/default-py39' not found.

I've solved it by using another virtual environment in python3.8 and everything when smoothly.

Here are my python environments specs' (via pip):

$ sdiff <(~/.virtualenvs/coiled/bin/pip list) <(~/.virtualenvs/coiled3.8/bin/pip list) 
Package           Version                   Package           Version
----------------- --------                  ----------------- --------
aiobotocore       1.2.0                     aiobotocore       1.2.0
aiohttp           3.7.3                     aiohttp           3.7.3
aioitertools      0.7.1                     aioitertools      0.7.1
async-timeout     3.0.1                     async-timeout     3.0.1
attrs             20.3.0                    attrs             20.3.0
backcall          0.2.0                     backcall          0.2.0
bokeh             2.2.3                     bokeh             2.2.3
boto3             1.16.52                   boto3             1.16.52
botocore          1.19.52                   botocore          1.19.52
chardet           3.0.4                     chardet           3.0.4
click             7.1.2                     click             7.1.2
cloudpickle       1.6.0                     cloudpickle       1.6.0
coiled            0.0.33                    coiled            0.0.33
cycler            0.10.0                    cycler            0.10.0
dask              2021.1.1                  dask              2021.1.1
decorator         4.4.2                     decorator         4.4.2
distributed       2021.1.1                  distributed       2021.1.1
fsspec            0.8.5                     fsspec            0.8.5
HeapDict          1.0.1                     HeapDict          1.0.1
idna              3.1                       idna              3.1
ipython           7.19.0                    ipython           7.19.0
ipython-genutils  0.2.0                     ipython-genutils  0.2.0
jedi              0.18.0                    jedi              0.18.0
Jinja2            2.11.2                    Jinja2            2.11.2
jmespath          0.10.0                    jmespath          0.10.0
kiwisolver        1.3.1                     kiwisolver        1.3.1
locket            0.2.1                     locket            0.2.1
MarkupSafe        1.1.1                     MarkupSafe        1.1.1
matplotlib        3.3.3                     matplotlib        3.3.3
msgpack           1.0.2                     msgpack           1.0.2
multidict         5.1.0                     multidict         5.1.0
numpy             1.19.5                    numpy             1.19.5
packaging         20.8                      packaging         20.8
pandas            1.2.1                     pandas            1.2.1
parso             0.8.1                     parso             0.8.1
partd             1.1.0                     partd             1.1.0
pexpect           4.8.0                     pexpect           4.8.0
pickleshare       0.7.5                     pickleshare       0.7.5
Pillow            8.1.0                     Pillow            8.1.0
pip               21.0                      pip               21.0
prompt-toolkit    3.0.14                    prompt-toolkit    3.0.14
psutil            5.8.0                     psutil            5.8.0
ptyprocess        0.7.0                     ptyprocess        0.7.0
Pygments          2.7.4                     Pygments          2.7.4
pyparsing         2.4.7                     pyparsing         2.4.7
python-dateutil   2.8.1                     python-dateutil   2.8.1
pytz              2020.5                    pytz              2020.5
PyYAML            5.4.1                     PyYAML            5.4.1
s3fs              0.5.2                     s3fs              0.5.2
s3transfer        0.3.4                     s3transfer        0.3.4
scipy             1.6.0                       <
setuptools        49.1.3                    setuptools        49.1.3
six               1.15.0                    six               1.15.0
sortedcontainers  2.3.0                     sortedcontainers  2.3.0
tblib             1.7.0                     tblib             1.7.0
toolz             0.11.1                    toolz             0.11.1
tornado           6.1                       tornado           6.1
traitlets         5.0.5                     traitlets         5.0.5
typing-extensions 3.7.4.3                   typing-extensions 3.7.4.3
urllib3           1.26.2                    urllib3           1.26.2
wcwidth           0.2.5                     wcwidth           0.2.5
wrapt             1.12.1                    wrapt             1.12.1
yarl              1.6.3                     yarl              1.6.3
zict              2.0.0                     zict              2.0.0

Quick start indications for software environment

In the Software pane, currently user get those indications for Pip:

The second snippets' software parameter can be fixed for Pip and Docker to use the proper environment when creating a cluster.

I spent some time trying to find a way to programmatically get, in a new interpreter, a coiled.Cluster which was allocated in a previous one. I found the solution with the API documentation in a few minutes.

---

I am spreading the words regarding the beta to some friends in ML/Engineering and may report some issues or feedback regarding it when I have some time (I can't give guarantee).

tl;dr: To me, it looks like a proper kick-off! 🙂

[FabioRosado]

Hello, @jjerphan thank you for reporting the issue, we recently bumped coiled to 0.0.33, and we don't have a default configuration that uses python 3.9 yet.

If you use conda you can specify which python version to use conda create -n coiled python-3.8 or use coiled install <software env> to create a new conda env and install all the dependencies (and versions) that the software env has. Note that version mismatch might throw some errors, so it's a good idea to use the same versions on your machine (client) and the scheduler/worker(cloud)

I'm glad that you are enjoying exploring coiled!

[jjerphan]

Thank for you response. It makes sense as most people rely on conda to manage their environments.

I'll let you know about my exploration. 🙂

[mrocklin]

It's worth noting that Dask itself is actually a little bit behind on 3.9 support. After Julien mentioned this in a conversation the other day I prodded this PR. https://github.com/dask/distributed/pull/4234

If anyone wants to investigate what is happening there that would be welcome.

On Tue, Jan 26, 2021 at 3:17 AM Julien Jerphanion notifications@github.com wrote:

Thank for you response. It makes sense as most people rely on conda to manage their environments.

I'll let you know about my exploration. 🙂

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/104#issuecomment-767476655, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTGVH6Y4FU7FLK2J25TS32QGFANCNFSM4WTCQS7A .

[jrbourbeau]

Thanks for the feedback @jjerphan!

cc @dantheman39 regarding the software environment quickstart

[necaris]

cc @meimeisuns :point_up: :smile:

[jjerphan]

Hi all!

I imagine Coiled has progressed a lot since then and that we can close this issue? 🙂

[dchudz]

Indeed! Thanks for the early feedback, and for circling back. I do think Coiled is a lot better now. :-)