DMLC_PS_ROOT_URI using hostname failed in distributed training

jzp1025 commented 6 years ago

Description

DMLC_PS_ROOT_URI is ip or hostname , but when use the hostname instead of ip , it reports "bind failed" in the src/van.cc

Environment info (Required)

----------Python Info----------
('Version      :', '2.7.12')
('Compiler     :', 'GCC 5.4.0 20160609')
('Build        :', ('default', 'Dec  4 2017 14:50:18'))
('Arch         :', ('64bit', 'ELF'))
------------Pip Info-----------
('Version      :', '10.0.1')
('Directory    :', '/usr/local/lib/python2.7/dist-packages/pip')
----------MXNet Info-----------
('Version      :', '1.0.0')
('Directory    :', '/home/tusimple/incubator-mxnet/python/mxnet')
Hashtag not found. Not installed from pre-built package.
----------System Info----------
('Platform     :', 'Linux-4.4.0-130-generic-x86_64-with-Ubuntu-16.04-xenial')
('system       :', 'Linux')
('node         :', 'tusimple-System-Product-Name')
('release      :', '4.4.0-130-generic')
('version      :', '#156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018')
----------Hardware Info----------
('machine      :', 'x86_64')
('processor    :', 'x86_64')
Architecture:          x86_64
CPU 运行模式：    32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
每个核的线程数：2
每个座的核数：  4
Socket(s):             1
NUMA 节点：         1
厂商 ID：           GenuineIntel
CPU 系列：          6
型号：              94
Model name:            Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
步进：              3
CPU MHz：             4200.156
CPU max MHz:           4200.0000
CPU min MHz:           800.0000
BogoMIPS:              8015.57
虚拟化：           VT-x
L1d 缓存：          32K
L1i 缓存：          32K
L2 缓存：           256K
L3 缓存：           8192K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0007 sec, LOAD: 1.5099 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.2137 sec, LOAD: 3.8772 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.2147 sec, LOAD: 1.0284 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0100 sec, LOAD: 0.7743 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0097 sec, LOAD: 2.1807 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0098 sec, LOAD: 1.6313 sec.

Package used (Python/R/Scala/Julia): python

Build info (Required if built from source)

gcc

MXNet commit hash: 3df9bf802021d5aa67c609c6736acee94aaf3a48

Build config: the same as doc https://mxnet.apache.org/install/index.html?platform=Linux&language=Python&processor=CPU

Error Message:

(Paste the complete error message, including stack trace.)

[17:46:11] /home/tusimple/incubator-mxnet/dmlc-core/include/dmlc/./logging.h:308: [17:46:11] src/van.cc:76: Check failed: (mynode.port) != (-1) bind failed

Stack trace returned 10 entries: [bt] (0) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f1a283f624c] [bt] (1) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN2ps3Van5StartEv+0x91f) [0x7f1a2af45b8f] [bt] (2) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN2ps6ZMQVan5StartEv+0x4a) [0x7f1a2af504fa] [bt] (3) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN2ps10Postoffice5StartEPKcb+0x1e9) [0x7f1a2af42119] [bt] (4) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN5mxnet7kvstore11KVStoreDist9RunServerERKSt8functionIFviRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEE+0x1c5) [0x7f1a2aee1c35] [bt] (5) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(MXKVStoreRunServer+0x4b) [0x7f1a2ae629db] [bt] (6) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f1a41450e40] [bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f1a414508ab] [bt] (8) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7f1a416603df] [bt] (9) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7f1a41664d82]

Traceback (most recent call last): File "train_mnist.py", line 25, in from common import find_mxnet, fit File "/home/tusimple/incubator-mxnet/example/image-classification/common/find_mxnet.py", line 20, in import mxnet as mx File "/home/tusimple/incubator-mxnet/example/image-classification/mxnet/init.py", line 56, in from . import kvstore_server File "/home/tusimple/incubator-mxnet/example/image-classification/mxnet/kvstore_server.py", line 85, in _init_kvstore_server_module() File "/home/tusimple/incubator-mxnet/example/image-classification/mxnet/kvstore_server.py", line 82, in _init_kvstore_server_module server.run() File "/home/tusimple/incubator-mxnet/example/image-classification/mxnet/kvstore_server.py", line 73, in run check_call(_LIB.MXKVStoreRunServer(self.handle, _ctrl_proto(self._controller()), None)) File "/home/tusimple/incubator-mxnet/example/image-classification/mxnet/base.py", line 146, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [17:46:11] src/van.cc:76: Check failed: (mynode.port) != (-1) bind failed

Stack trace returned 10 entries: [bt] (0) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f1a283f624c] [bt] (1) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN2ps3Van5StartEv+0x91f) [0x7f1a2af45b8f] [bt] (2) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN2ps6ZMQVan5StartEv+0x4a) [0x7f1a2af504fa] [bt] (3) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN2ps10Postoffice5StartEPKcb+0x1e9) [0x7f1a2af42119] [bt] (4) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(_ZN5mxnet7kvstore11KVStoreDist9RunServerERKSt8functionIFviRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEE+0x1c5) [0x7f1a2aee1c35] [bt] (5) /home/tusimple/incubator-mxnet/example/image-classification/mxnet/libmxnet.so(MXKVStoreRunServer+0x4b) [0x7f1a2ae629db] [bt] (6) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f1a41450e40] [bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f1a414508ab] [bt] (8) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7f1a416603df] [bt] (9) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7f1a41664d82]

Minimum reproducible example

1 scheduler 1 server 1 worker

Steps to reproduce

(Paste the commands you ran that produced the error.)

1.export DMLC_PS_ROOT_URI=tusimple-System-Product-Name; export DMLC_ROLE=scheduler; export DMLC_PS_ROOT_PORT=9001; export DMLC_NUM_WORKER=1; export DMLC_NUM_SERVER=1; 2.python train_mnist.py

What have you tried to solve it?

1.i replaced the DMLC_PS_ROOT_URI with ip and it works well

apeforest commented 6 years ago

@mxnet-label-bot could you please add [Backend] here?

xcgoner commented 6 years ago

I think this is actually an issue of zmq, used in ps-lite. Zmq can only take ip address to find network interface. Using hostname will fail. Please take a look at the discussion here: https://stackoverflow.com/questions/6024003/why-doesnt-zeromq-work-on-localhost

zhipeng93 commented 5 years ago

thanks!

pinaraws commented 5 years ago

@mxnet-label-bot add[Distributed]

apache / mxnet