search

dmlc / dmlc-core

A common bricks library for building scalable and portable distributed machine learning.
Apache License 2.0
864 stars 520 forks source link

Update yarn.py #575

Closed wendycwong closed 5 years ago

wendycwong commented 5 years ago

Two bugs discovered by Michal M.

wendycwong commented 5 years ago

I was trying to run xgboost 0.82 off hadoop/yarn. It needs to use python3 specified in kubernetes.py.

[1]+ Exit 1 nohup ./dmlc-submit --cluster=yarn --num-workers=4 --worker-memory='50g' /home/wendy/xgboost_reproducibility/xgboost/xgboost higgs_all.aws.conf > HiggsRun1 (wendy_env_35) wendy@mr-0xd2:~/xgboost_reproducibility/xgboost/dmlc-core/tracker$ cat HiggsRun1 2019-10-21 10:31:31,203 INFO start listen on 172.16.2.182:9094 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 Traceback (most recent call last): File "./dmlc-submit", line 9, in submit.main() File "/home/wendy/xgboost_reproducibility/xgboost/dmlc-core/tracker/dmlc_tracker/submit.py", line 48, in main yarn.submit(args) File "/home/wendy/xgboost_reproducibility/xgboost/dmlc-core/tracker/dmlc_tracker/yarn.py", line 130, in submit pscmd=(' '.join([YARN_BOOT_PY] + args.command))) File "/home/wendy/xgboost_reproducibility/xgboost/dmlc-core/tracker/dmlc_tracker/tracker.py", line 423, in submit fun_submit(nworker, nserver, envs) File "/home/wendy/xgboost_reproducibility/xgboost/dmlc-core/tracker/dmlc_tracker/yarn.py", line 124, in yarn_submit_pass submit_thread.append(yarn_submit(args, nworker, nserver, pass_env)) File "/home/wendy/xgboost_reproducibility/xgboost/dmlc-core/tracker/dmlc_tracker/yarn.py", line 49, in yarn_submit out = out.split('\n')[0].split() TypeError: a bytes-like object is required, not 'str'

hcho3 commented 5 years ago

@wendycwong Can you review #564? It addresses the same bug and supports both Python 2 and 3.

wendycwong commented 5 years ago

Yes, you are right! That is the same fix. I approved it.

On Oct 24, 2019, at 4:40 AM, Philip Hyunsu Cho notifications@github.com wrote:

@wendycwong https://github.com/wendycwong Can you review #564 https://github.com/dmlc/dmlc-core/pull/564? It addresses the same bug and supports both Python 2 and 3.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmlc/dmlc-core/pull/575?email_source=notifications&email_token=ABXFRYDXMBSG52DP4W7SFL3QQGCTLA5CNFSM4JCMLMHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECEXETY#issuecomment-545878607, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXFRYH6RXW66FOESZMEN43QQGCTLANCNFSM4JCMLMHA.

hcho3 commented 5 years ago

@wendycwong Thanks for your review. I just merged #564 so closing this one now.