Open qiuxin2012 opened 1 year ago
executor could not find hadoop commad, user should add soft link to /usr/local/bin
. Like
sudo ln -s $HADOOP_HOME/bin/hadoop /usr/local/bin
sudo ln -s $HADOOP_HOME/bin/hdfs /usr/local/bin
sudo ln -s $HADOOP_HOME/libexec /usr/local
get another error:
****************************Usage Error************************
get: `/tmp/auto_lstm/auto_lstm/train_func_0d583_00039/best.ckpt': File exists
****************************Call Stack*************************
Traceback (most recent call last):
File "lstm.py", line 82, in <module>
n_sampling=40)
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/chronos/autots/model/base_automodel.py", line 101, in fit
self.best_model = self.auto_est._get_best_automl_model()
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/auto_estimator.py", line 244, in _get_best_automl_model
self.best_trial = self.searcher.get_best_trial()
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 222, in get_best_trial
return self.get_best_trials(k=1)[0]
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 234, in get_best_trials
return [self._make_trial_output(t) for t in best_trials]
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 234, in <listcomp>
return [self._make_trial_output(t) for t in best_trials]
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 239, in _make_trial_output
get_ckpt_hdfs(self.remote_dir, model_path)
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/utils.py", line 92, in get_ckpt_hdfs
process(cmd)
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/utils.py", line 39, in process
invalidInputError(False, err)
File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/dllib/utils/log4Error.py", line 33, in invalidInputError
raise RuntimeError(errMsg)
RuntimeError: get: `/tmp/auto_lstm/auto_lstm/train_func_0d583_00039/best.ckpt': File exists
Stopping orca context
get another error:
****************************Usage Error************************ get: `/tmp/auto_lstm/auto_lstm/train_func_0d583_00039/best.ckpt': File exists ****************************Call Stack************************* Traceback (most recent call last): File "lstm.py", line 82, in <module> n_sampling=40) File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/chronos/autots/model/base_automodel.py", line 101, in fit self.best_model = self.auto_est._get_best_automl_model() File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/auto_estimator.py", line 244, in _get_best_automl_model self.best_trial = self.searcher.get_best_trial() File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 222, in get_best_trial return self.get_best_trials(k=1)[0] File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 234, in get_best_trials return [self._make_trial_output(t) for t in best_trials] File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 234, in <listcomp> return [self._make_trial_output(t) for t in best_trials] File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 239, in _make_trial_output get_ckpt_hdfs(self.remote_dir, model_path) File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/utils.py", line 92, in get_ckpt_hdfs process(cmd) File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/orca/automl/search/utils.py", line 39, in process invalidInputError(False, err) File "/home/cpx/miniconda3/envs/xin-chronos/lib/python3.7/site-packages/bigdl/dllib/utils/log4Error.py", line 33, in invalidInputError raise RuntimeError(errMsg) RuntimeError: get: `/tmp/auto_lstm/auto_lstm/train_func_0d583_00039/best.ckpt': File exists Stopping orca context
this error will happen when driver is running on the same machine with executor. Because driver and executor is using the same local tmp dir.
init spark on yarn with ray, automl yarn-client mode couldn't find hadoop command, when hadoop is not in default linux PATH. Apache hadoop has this problem, while CDH does't have.