search

dask / dask-yarn

Deploy dask on YARN clusters
http://yarn.dask.org
BSD 3-Clause "New" or "Revised" License
69 stars 41 forks source link

Use the full Python path when using archived environment #115

Closed jcrist closed 4 years ago

jcrist commented 4 years ago

Previously, if an archived environment was provided, we'd generate the following script for running a dask worker:

source environment/bin/activate
dask-yarn services worker

This works fine on most systems, but for some reason fails on some systems. The failure is odd - the dask-yarn CLI is properly found, but the dask_yarn library fails to be on Python path. This may be due to the shell not being rehashed properly, so the wrong python is used, I'm not sure. It's likely a bug in conda-pack.

Either way, specifying the full path to the proper Python fixes things, so we use that here. We now generate the following script:

source environment/bin/activate
environment/bin/python -m dask_yarn.cli services worker

This should work in all cases, even in the presence of a conda-pack bug (which we should still fix).

Fixes #114, fixes #111.

DanRunfola commented 4 years ago

Thanks for the help and exploration into this, Jim.