search

apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.78k stars 6.79k forks source link

MXNDArraySyncCopyToCPU() hangs #19994

Open nagyt2 opened 3 years ago

nagyt2 commented 3 years ago

Description

After running our python program several hours it comes to a point when it just hangs. I was able to attach to the process with both strace and gdb. Strace showed that the process waits on a futex: 

strace: Process 5567 attached
futex(0x556f2ffce8, FUTEX_WAIT_PRIVATE, 0, NULL

Then the bactrace in gdb clearly shows that the issue comes from MXNDArraySyncCopyToCPU (Please see lines 4-7):

(gdb) bt
#0  0x0000007fa49bf2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556f2ffce8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  0x0000007fa49bf2a4 in __pthread_cond_wait_common (abstime=0x0, mutex=0x556f2ffc90, cond=0x556f2ffcc0) at pthread_cond_wait.c:502
#2  0x0000007fa49bf2a4 in __pthread_cond_wait (cond=0x556f2ffcc0, mutex=0x556f2ffc90) at pthread_cond_wait.c:655
#3  0x0000007f9f31fb78 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x0000007f747f06ec in mxnet::engine::ThreadedEngine::WaitForVar(mxnet::engine::Var*) () at /home/ubuntu/.cache/pypoetry/virtualenvs/g-R0-ACE-i-py3.7/lib/python3.7/site-packages/mxnet/libmxnet.so
#5  0x0000007f7406f1d0 in mxnet::NDArray::WaitToWrite() const () at /home/ubuntu/.cache/pypoetry/virtualenvs/g-R0-ACE-i-py3.7/lib/python3.7/site-packages/mxnet/libmxnet.so
#6  0x0000007f741ba700 in mxnet::NDArray::SyncCopyToCPU(void*, unsigned long) const () at /home/ubuntu/.cache/pypoetry/virtualenvs/g-R0-ACE-i-py3.7/lib/python3.7/site-packages/mxnet/libmxnet.so
#7  0x0000007f748680bc in MXNDArraySyncCopyToCPU () at /home/ubuntu/.cache/pypoetry/virtualenvs/g-R0-ACE-i-py3.7/lib/python3.7/site-packages/mxnet/libmxnet.so
#8  0x0000007f9ef75d28 in ffi_call_SYSV () at /usr/lib/aarch64-linux-gnu/libffi.so.6
#9  0x0000007f9ef76698 in ffi_call () at /usr/lib/aarch64-linux-gnu/libffi.so.6
#10 0x0000007f94cc5a60 in _call_function_pointer (argcount=3, resmem=0x7e4e8be2d0, restype=<optimized out>, atypes=<optimized out>, avalues=0x7fde292fe0, pProc=0x3, flags=-1798590016) at /tmp/python-build.20200412011356.1736/Python-3.7.7/Modules/_ctypes/callproc.c:829
#11 0x0000007f94cc5a60 in _ctypes_callproc (pProc=0x3, pProc@entry=0x7fa45497b8, argtuple=argtuple@entry=
    (<c_void_p at remote 0x7e65fe3cb0>, <c_void_p(_arr=<numpy.ndarray at remote 0x7e4e98e030>) at remote 0x7e65fa8f80>, <c_ulong at remote 0x7e65fe3680>), flags=-1798590016, argtypes=argtypes@entry=0x0, restype=0x0, 
    restype@entry=<_ctypes.PyCSimpleType at remote 0x556d624c10>, checker=<unknown at remote 0x7fa4e1a000>) at /tmp/python-build.20200412011356.1736/Python-3.7.7/Modules/_ctypes/callproc.c:1186
#12 0x0000007f94cbb280 in PyCFuncPtr_call (self=0x7eb00cbef0, inargs=<optimized out>, kwds=<unknown at remote 0x7fa4e1a000>) at /tmp/python-build.20200412011356.1736/Python-3.7.7/Modules/_ctypes/_ctypes.c:4025
#13 0x0000007fa4bfbe1c in _PyObject_FastCallKeywords (callable=<_FuncPtr(__name__='MXNDArraySyncCopyToCPU') at remote 0x7eb00cbef0>, stack=<optimized out>, nargs=3, kwnames=<optimized out>) at Objects/call.c:199
#14 0x0000007fa4bd0af4 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4619
#15 0x0000007fa4bd0af4 in _PyEval_EvalFrameDefault (f=<unknown at remote 0x7e660e82e8>, throwflag=<optimized out>) at Python/ceval.c:3093
#16 0x0000007fa4bc8cec in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at Objects/call.c:283
#17 0x0000007fa4bcfc18 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#18 0x0000007fa4bcfc18 in _PyEval_EvalFrameDefault (f=<unknown at remote 0x55a4348708>, throwflag=<optimized out>) at Python/ceval.c:3110
#19 0x0000007fa4cf2140 in _PyEval_EvalCodeWithName (_co=<code at remote 0x7f2c029030>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=13, kwnames=0xd, kwargs=0x7f2c0a0930, kwcount=0, kwstep=<optimized out>, 
    kwstep@entry=1, defs=0x7f2c026a68, defcount=<optimized out>, kwdefs=0x0, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at Python/ceval.c:3930
#20 0x0000007fa4bfb6fc in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Objects/call.c:433
#21 0x0000007fa4bd04d8 in call_function (kwnames=('force_reshape',), oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#22 0x0000007fa4bd04d8 in _PyEval_EvalFrameDefault (f=<unknown at remote 0x55a43484a8>, throwflag=<optimized out>) at Python/ceval.c:3139
#23 0x0000007fa4cf2140 in _PyEval_EvalCodeWithName (_co=<code at remote 0x7f2c0a0930>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=3, kwnames=0x556d349770, kwargs=0x7fa4e1a000 <langinfo_constants+592>, kwcount=367827125776, kwstep=<optimized out>, kwstep@entry=1, defs=0x7f2c025728, defcount=<optimized out>, kwdefs=0x0, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at Python/ceval.c:3930
#24 0x0000007fa4bfb6fc in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Objects/call.c:433
#25 0x0000007fa4bcfdac in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#26 0x0000007fa4bcfdac in _PyEval_EvalFrameDefault (f=<unknown at remote 0x55a4266188>, throwflag=<optimized out>) at Python/ceval.c:3124
#27 0x0000007fa4bc8cec in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=3, globals=<optimized out>) at Objects/call.c:283
#28 0x0000007fa4bcfc18 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#29 0x0000007fa4bcfc18 in _PyEval_EvalFrameDefault (f=<unknown at remote 0x55aba447b8>, throwflag=<optimized out>) at Python/ceval.c:3110
#30 0x0000007fa4cf2140 in _PyEval_EvalCodeWithName (_co=<code at remote 0x7f2c0a0540>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=2, kwnames=0x556d349770, kwargs=0x7fa4e1a000 <langinfo_constants+592>, kwcount=367951890720, kwstep=<optimized out>, kwstep@entry=1, defs=0x7f2c09dfe8, defcount=<optimized out>, kwdefs=0x0, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at Python/ceval.c:3930
#31 0x0000007fa4bfb6fc in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Objects/call.c:433
#32 0x0000007fa4bcfc18 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#33 0x0000007fa4bcfc18 in _PyEval_EvalFrameDefault (f=<unknown at remote 0x7e4f5f8ee8>, throwflag=<optimized out>) at Python/ceval.c:3110
#34 0x0000007fa4bc8cec in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#35 0x0000007fa4bcfc18 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#36 0x0000007fa4bcfc18 in _PyEval_EvalFrameDefault (f=<unknown at remote 0x7e4e989488>, throwflag=<optimized out>) at Python/ceval.c:3110
#37 0x0000007fa4bc8cec in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at Objects/call.c:283
#38 0x0000007fa4bcfc18 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#39 0x0000007fa4bcfc18 in _PyEval_EvalFrameDefault (f=<unknown at remote 0x556f2c8388>, throwflag=<optimized out>) at Python/ceval.c:3110
#40 0x0000007fa4bc8cec in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at Objects/call.c:283
#41 0x0000007fa4bcfc18 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#42 0x0000007fa4bcfc18 in _PyEval_EvalFrameDefault (f=<unknown at remote 0x7fa41acee8>, throwflag=<optimized out>) at Python/ceval.c:3110
#43 0x0000007fa4bc8cec in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at Objects/call.c:283
#44 0x0000007fa4bcfdac in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#45 0x0000007fa4bcfdac in _PyEval_EvalFrameDefault (f=<unknown at remote 0x556d394908>, throwflag=<optimized out>) at Python/ceval.c:3124
#46 0x0000007fa4bc8cec in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at Objects/call.c:283
#47 0x0000007fa4bcfdac in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#48 0x0000007fa4bcfdac in _PyEval_EvalFrameDefault (f=<unknown at remote 0x7fa443f738>, throwflag=<optimized out>) at Python/ceval.c:3124
#49 0x0000007fa4cf2140 in _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fa443f5d0>, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py') at remote 0x7fa44dd510>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa4598d10>, '__file__': '/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py', '__cached__': None, 'logging': <module at remote 0x7fa43ab590>, 'sys': <module at remote 0x7fa4592d70>, 'os': <module at remote 0x7fa44d8b30>, 'parse_main_args': <function at remote 0x7fa4298680>, 'App': <EnumMeta(_generate_next_value_=<function at remote 0x7fa430c4d0>, __module__='a.g.project_structure', __init__=<function at remote 0x7fa41d18c0>, __str__=<function at remote 0x7fa41f8dd0>, __repr__=<function at remote 0x7fa41f8e60>, __doc__='An enumeration.', _member_names_=['AR', 'STADIUM', 'SKIDATA', 'AVATARION', 's', 'CONFERENCE', 'GANTNER', 'JETSON'], _member_map_={'AR': <App(_value_=...(truncated), locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py') at remote 0x7fa44dd510>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa4598d10>, '__file__': '/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py', '__cached__': None, 'logging': <module at remote 0x7fa43ab590>, 'sys': <module at remote 0x7fa4592d70>, 'os': <module at remote 0x7fa44d8b30>, 'parse_main_args': <function at remote 0x7fa4298680>, 'App': <EnumMeta(_generate_next_value_=<function at remote 0x7fa430c4d0>, __module__='a.g.project_structure', __init__=<function at remote 0x7fa41d18c0>, __str__=<function at remote 0x7fa41f8dd0>, __repr__=<function at remote 0x7fa41f8e60>, __doc__='An enumeration.', _member_names_=['AR', 'STADIUM', 'SKIDATA', 'AVATARION', 's', 'CONFERENCE', 'GANTNER', 'JETSON'], _member_map_={'AR': <App(_value_=...(truncated), args=args@entry=0x0, argcount=548217913904, argcount@entry=0, kwnames=0x7fa44dcd90, kwnames@entry=0x0, kwargs=0x7fa4d29890 <PyRun_FileExFlags+184>, kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=<optimized out>, kwstep@entry=2, defs=defs@entry=0x0, defcount=<optimized out>, 
    defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=<optimized out>, closure@entry=0x0, name=<optimized out>, name@entry=0x0, qualname=<optimized out>, qualname@entry=0x0) at Python/ceval.c:3930
#50 0x0000007fa4cf22e0 in PyEval_EvalCodeEx (_co=_co@entry=<code at remote 0x7fa443f5d0>, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py') at remote 0x7fa44dd510>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa4598d10>, '__file__': '/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py', '__cached__': None, 'logging': <module at remote 0x7fa43ab590>, 'sys': <module at remote 0x7fa4592d70>, 'os': <module at remote 0x7fa44d8b30>, 'parse_main_args': <function at remote 0x7fa4298680>, 'App': <EnumMeta(_generate_next_value_=<function at remote 0x7fa430c4d0>, __module__='a.g.project_structure', __init__=<function at remote 0x7fa41d18c0>, __str__=<function at remote 0x7fa41f8dd0>, __repr__=<function at remote 0x7fa41f8e60>, __doc__='An enumeration.', _member_names_=['AR', 'STADIUM', 'SKIDATA', 'AVATARION', 's', 'CONFERENCE', 'GANTNER', 'JETSON'], _member_map_={'AR': <App(_value_=...(truncated), locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py') at remote 0x7fa44dd510>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa4598d10>, '__file__': '/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py', '__cached__': None, 'logging': <module at remote 0x7fa43ab590>, 'sys': <module at remote 0x7fa4592d70>, 'os': <module at remote 0x7fa44d8b30>, 'parse_main_args': <function at remote 0x7fa4298680>, 'App': <EnumMeta(_generate_next_value_=<function at remote 0x7fa430c4d0>, __module__='a.g.project_structure', __init__=<function at remote 0x7fa41d18c0>, __str__=<function at remote 0x7fa41f8dd0>, __repr__=<function at remote 0x7fa41f8e60>, __doc__='An enumeration.', _member_names_=['AR', 'STADIUM', 'SKIDATA', 'AVATARION', 's', 'CONFERENCE', 'GANTNER', 'JETSON'], _member_map_={'AR': <App(_value_=...(truncated), args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at Python/ceval.c:3959
#51 0x0000007fa4cf231c in PyEval_EvalCode (co=co@entry=<code at remote 0x7fa443f5d0>, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/a/g/bin/../components/g/src---Type <return> to continue, or q <return> to quit---
/py/a/g/main.py') at remote 0x7fa44dd510>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa4598d10>, '__file__': '/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py', '__cached__': None, 'logging': <module at remote 0x7fa43ab590>, 'sys': <module at remote 0x7fa4592d70>, 'os': <module at remote 0x7fa44d8b30>, 'parse_main_args': <function at remote 0x7fa4298680>, 'App': <EnumMeta(_generate_next_value_=<function at remote 0x7fa430c4d0>, __module__='a.g.project_structure', __init__=<function at remote 0x7fa41d18c0>, __str__=<function at remote 0x7fa41f8dd0>, __repr__=<function at remote 0x7fa41f8e60>, __doc__='An enumeration.', _member_names_=['AR', 'STADIUM', 'SKIDATA', 'AVATARION', 's', 'CONFERENCE', 'GANTNER', 'JETSON'], _member_map_={'AR': <App(_value_=...(truncated), locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py') at remote 0x7fa44dd510>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa4598d10>, '__file__': '/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py', '__cached__': None, 'logging': <module at remote 0x7fa43ab590>, 'sys': <module at remote 0x7fa4592d70>, 'os': <module at remote 0x7fa44d8b30>, 'parse_main_args': <function at remote 0x7fa4298680>, 'App': <EnumMeta(_generate_next_value_=<function at remote 0x7fa430c4d0>, __module__='a.g.project_structure', __init__=<function at remote 0x7fa41d18c0>, __str__=<function at remote 0x7fa41f8dd0>, __repr__=<function at remote 0x7fa41f8e60>, __doc__='An enumeration.', _member_names_=['AR', 'STADIUM', 'SKIDATA', 'AVATARION', 's', 'CONFERENCE', 'GANTNER', 'JETSON'], _member_map_={'AR': <App(_value_=...(truncated)) at Python/ceval.c:524
#52 0x0000007fa4d29890 in run_mod (arena=0x7fa44dcd90, flags=0x7fde294770, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': ... at Python/pythonrun.c:1035
#53 0x0000007fa4d29890 in PyRun_FileExFlags (fp=fp@entry=0x556d394ec0, filename_str=filename_str@entry=0x7fa448be50 "/home/ubuntu/a/g/bin/../components/g/src/py/a/g/main.py", start=start@entry=257,... at Python/pythonrun.c:429
#55 0x0000007fa4d29f18 in PyRun_AnyFileExFlags (fp=fp@entry=0x556d394ec0, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fde294770) at Python/pythonrun.c:84
#56 0x0000007fa4d4d01c in pymain_run_file (p_cf=0x7fde294770, filename=<optimized out>, fp=0x556d394ec0) at Modules/main.c:456
#57 0x0000007fa4d4d01c in pymain_run_filename (cf=0x7fde294770, pymain=0x7fde294848) at Modules/main.c:1635
#58 0x0000007fa4d4d01c in pymain_run_python (pymain=0x7fde294848) at Modules/main.c:2896
#59 0x0000007fa4d4d01c in pymain_main (pymain=0x7fde294848, pymain@entry=0x7fde294858) at Modules/main.c:3057
#60 0x0000007fa4d4d2cc in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:3092
#61 0x0000007fa49fe720 in __libc_start_main (main=0x0, argc=0, argv=0x0, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:310
#62 0x000000556131b908 in _start ()

Error Message

There is no error message, the application just hangs

To Reproduce

The issue comes sporadically sometimes after hours, sometimes after days, it's hard to reproduce

Environment

The runtime environment is based on the Jetson Nano box from NVidia. Processor is armv8, we have 4GB RAM. mxnet 1.6 w/ CUDA 10.0 are used.

Please note that the PYTHON version reported by diagnose.py differs from the one we are using running our application. (We are using pyenv, not the distribution delivered python package). We run the application using Python 3.7.7.

Environment Information ``` ----------Python Info---------- Version : 3.6.9 Compiler : GCC 8.4.0 Build : ('default', 'Jan 26 2021 15:33:00') Arch : ('64bit', 'ELF') ------------Pip Info----------- Version : 9.0.1 Directory : /usr/lib/python3/dist-packages/pip ----------MXNet Info----------- No MXNet installed. ----------System Info---------- Platform : Linux-4.9.140-tegra-aarch64-with-Ubuntu-18.04-bionic system : Linux node : sc1 release : 4.9.140-tegra version : #1 SMP PREEMPT Mon Dec 9 22:47:42 PST 2019 ----------Hardware Info---------- machine : aarch64 processor : aarch64 Architecture: aarch64 Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Vendor ID: ARM Model: 1 Model name: Cortex-A57 Stepping: r1p1 CPU max MHz: 1479.0000 CPU min MHz: 102.0000 BogoMIPS: 38.40 L1d cache: 32K L1i cache: 48K L2 cache: 2048K Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 ----------Network Test---------- Setting timeout: 10 Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0020 sec, LOAD: 0.5257 sec. Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0012 sec, LOAD: 0.2078 sec. Error open Gluon Tutorial(cn): https://zh.gluon.ai, , DNS finished in 0.0013172626495361328 sec. Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0007 sec, LOAD: 0.6361 sec. Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0099 sec, LOAD: 0.1532 sec. Error open Conda: https://repo.continuum.io/pkgs/free/, HTTP Error 403: Forbidden, DNS finished in 0.014477014541625977 sec. ----------Environment---------- ```

If I can provide any more information in order to hunt this bug down, please do not hesitate to let me know!

github-actions[bot] commented 3 years ago

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue. Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly. If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.