search

joblib / joblib-hadoop

Use Joblib in an Hadoop Cluster
Other
4 stars 1 forks source link

Joblib yarn test not working #10

Open rodfloripa opened 4 years ago

rodfloripa commented 4 years ago

I tried to run the joblib_yarn_parallel.py file and got an error(https://github.com/joblib/joblib-hadoop/blob/master/examples/joblib_yarn_parallel.py):

(/tmp/joblib_yarn_conda_env) [opc@brora1vudap003 joblib-hadoop]$ python test Traceback (most recent call last): File "test", line 13, in delayed(sqrt)(i2) for i in range(10))) File "/tmp/joblib_yarn_conda_env/lib/python3.5/site-packages/joblib/parallel.py", line 953, in call n_jobs = self._initialize_backend() File "/tmp/joblib_yarn_conda_env/lib/python3.5/site-packages/joblib/parallel.py", line 714, in _initialize_backend self._backend_args) File "/home/opc/rodney/joblib-hadoop/joblibhadoop/yarn/backend.py", line 37, in configure packages=self.packages, clear_env=self.clear_env) File "/home/opc/rodney/joblib-hadoop/joblibhadoop/yarn/pool.py", line 73, in init env='{}.zip'.format(os.path.join(TEMP_DIR, env))) TypeError: start() got an unexpected keyword argument 'env'

When I tried to use joblib locally it worked fine:

from joblib import Parallel, delayed from math import sqrt Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10)) [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

almeidajeff commented 3 years ago

I have the same problem using Anaconda3 on CDP.