apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.75k stars 6.8k forks source link

gluon.data.DataLoader when set num_workers>0, Exceptions always occurred #21084

Open CookingANut opened 2 years ago

CookingANut commented 2 years ago

Description

train_iter = gluon.data.DataLoader( mnist_train.transform_first(transformer), batch_size, shuffle=True, num_workers=4 )

when i set num_workers = 4 seems code can run through, but will appear below exception:

Exception ignored in: <function DataLoader.del at 0x7f1d7fe165e0> Traceback (most recent call last): File "/home/daemonh/miniconda3/envs/d2l/lib/python3.8/site-packages/mxnet/gluon/data/dataloader.py", line 720, in del AttributeError: 'NoneType' object has no attribute 'Pool' Exception ignored in: <function Pool.del at 0x7f1d7fe84670> Traceback (most recent call last): File "/home/daemonh/miniconda3/envs/d2l/lib/python3.8/multiprocessing/pool.py", line 268, in del File "/home/daemonh/miniconda3/envs/d2l/lib/python3.8/multiprocessing/queues.py", line 362, in put AttributeError: 'NoneType' object has no attribute 'dumps'

Does it need to focus or just common issue need to ignore, thanks you very much!

Environment

----------Python Info---------- Version : 3.8.13 Compiler : GCC 7.5.0 Build : ('default', 'Mar 28 2022 11:38:47') Arch : ('64bit', 'ELF') ----------MXNet Info----------- Version : 1.9.1 Directory : /home/daemonh/miniconda3/envs/d2l/lib/python3.8/site-packages/mxnet Commit hash file "/home/daemonh/miniconda3/envs/d2l/lib/python3.8/site-packages/mxnet/COMMIT_HASH" not found. Not installed from pre-built package or built from source. Library : ['/home/daemonh/miniconda3/envs/d2l/lib/python3.8/site-packages/mxnet/libmxnet.so'] Build features: ✔ CUDA ✔ CUDNN ✔ NCCL ✔ CUDA_RTC ✖ TENSORRT ✔ CPU_SSE ✔ CPU_SSE2 ✔ CPU_SSE3 ✖ CPU_SSE4_1 ✖ CPU_SSE4_2 ✖ CPU_SSE4A ✖ CPU_AVX ✖ CPU_AVX2 ✔ OPENMP ✖ SSE ✖ F16C ✖ JEMALLOC ✔ BLAS_OPEN ✖ BLAS_ATLAS ✖ BLAS_MKL ✖ BLAS_APPLE ✔ LAPACK ✔ MKLDNN ✔ OPENCV ✖ CAFFE ✖ PROFILER ✔ DIST_KVSTORE ✖ CXX14 ✖ INT64_TENSOR_SIZE ✔ SIGNAL_HANDLER ✖ DEBUG ✖ TVM_OP ----------System Info---------- Platform : Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.17 system : Linux release : 5.10.102.1-microsoft-standard-WSL2 version : #1 SMP Wed Mar 2 00:30:59 UTC 2022

github-actions[bot] commented 2 years ago

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue. Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly. If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

CookingANut commented 2 years ago

@apache/mxnet-committers