search

pangeo-data / helm-chart

Pangeo helm charts
https://pangeo-data.github.io/helm-chart/
21 stars 26 forks source link

WIP: compatible worker image #41

Closed rabernat closed 6 years ago

rabernat commented 6 years ago

I am trying to build a new worker image with compatible package versions with the latest notebook image (see #39).

Right now I am stuck on an obscure distributed error. When the script calls dask-worker, it gives the error:

$ dask-worker --nthreads 2 --no-bokeh --memory-limit 6GB --death-timeout 60
Traceback (most recent call last):
  File "/opt/conda/bin/dask-worker", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3095, in <module>
    @_call_aside
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3079, in _call_aside
    f(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3108, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 570, in _build_master
    ws.require(__requires__)
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 888, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 774, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack' distribution was not found and is required by distributed

I feel like the dask people will know how to solve this, so cc @mrocklin, @martindurant.

martindurant commented 6 years ago

Testing this in a local docker image, I find things work fine with distributed=1.21.8, 1.22.0 (conda) and 1.22.0 (pip) but not with that specific commit.

https://github.com/dask/distributed/pull/1927 changed the requirement name in distributed on May 2, because in pip it is "msgpack" and in conda "msgpack-python" - that doesn't explain the error, though. Presumably you get a different install path when working with source compared to pypi distributions.

mrocklin commented 6 years ago

Yes, I agree with @martindurant and recommend updating the dask/dask dask/distributed and dask/dask-kubernetes versions to latest release, 0.18.0, 1.22.0 and 0.4.0 respectively.

On Sun, Jun 17, 2018 at 9:28 AM, Martin Durant notifications@github.com wrote:

Testing this in a local docker image, I find things work fine with distributed=1.21.8, 1.22.0 (conda) and 1.22.0 (pip) but not with that specific commit.

dask/distributed#1927 https://github.com/dask/distributed/pull/1927 changed the requirement name in distributed on May 2, because in pip it is "msgpack" and in conda "msgpack-python" - that doesn't explain the error, though. Presumably you get a different install path when working with source compared to pypi distributions.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/helm-chart/pull/41#issuecomment-397879135, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszPpRD5-sG_UYFmXNQDPjWHG43T5mks5t9lmHgaJpZM4Uqsw0 .

rabernat commented 6 years ago

I now have what I believe to be a consistent set of versions. I am going to merge in order to get chartpress to build a new chart.

mrocklin commented 6 years ago

I recommend trying helm upgrade before trying helm install

On Sun, Jun 17, 2018 at 9:08 PM, Ryan Abernathey notifications@github.com wrote:

Merged #41 https://github.com/pangeo-data/helm-chart/pull/41.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/helm-chart/pull/41#event-1685193713, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszOMjexPka0_Q0v4KtUsSxmWrYnwPks5t9v12gaJpZM4Uqsw0 .