Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
For bugs or installation issues, please provide the following information.
The more information you provide, the more likely people will be able to help you.
An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
terminate called after throwing an instance of 'dmlc::Error'
what(): [15:13:11] src/engine/./threaded_engine.h:336: [15:13:11] src/operator/proposal.cu:496: Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
For bugs or installation issues, please provide the following information. The more information you provide, the more likely people will be able to help you.
Environment info
Operating System:ubuntu14.04
Compiler:
Package used (Python/R/Scala/Julia):
MXNet version:0.94
Or if installed from source:
MXNet commit hash (
git rev-parse HEAD
):If you are using python package, please provide
Python version and distribution:2.7
Error Message:
Please paste the full error message, including stack trace. `Namespace(dataset='PascalVOC', dataset_path='data/VOCdevkit', epoch=10, gpu=0, has_rpn=True, image_set='2007_test', network='vgg', prefix='model/e2e', proposal='rpn', root_path='data', shuffle=False, thresh=0.001, vis=False) {'ANCHOR_RATIOS': [0.5, 1, 2], 'ANCHOR_SCALES': [8, 16, 32], 'FIXED_PARAMS': ['conv1', 'conv2'], 'FIXED_PARAMS_SHARED': ['conv1', 'conv2', 'conv3', 'conv4', 'conv5'], 'IMAGE_STRIDE': 0, 'NUM_ANCHORS': 9, 'NUM_CLASSES': 2, 'PIXEL_MEANS': array([ 103.939, 116.779, 123.68 ]), 'RCNN_FEAT_STRIDE': 16, 'RPN_FEAT_STRIDE': 16, 'SCALES': [(600, 1000)], 'TEST': {'BATCH_IMAGES': 1, 'CXX_PROPOSAL': True, 'HAS_RPN': True, 'NMS': 0.3, 'PROPOSAL_MIN_SIZE': 16, 'PROPOSAL_NMS_THRESH': 0.7, 'PROPOSAL_POST_NMS_TOP_N': 2000, 'PROPOSAL_PRE_NMS_TOP_N': 20000, 'RPN_MIN_SIZE': 16, 'RPN_NMS_THRESH': 0.7, 'RPN_POST_NMS_TOP_N': 300, 'RPN_PRE_NMS_TOP_N': 6000}, 'TRAIN': {'ASPECT_GROUPING': True, 'BATCH_IMAGES': 2, 'BATCH_ROIS': 128, 'BBOX_MEANS': [0.0, 0.0, 0.0, 0.0], 'BBOX_NORMALIZATION_PRECOMPUTED': False, 'BBOX_REGRESSION_THRESH': 0.5, 'BBOX_STDS': [0.1, 0.1, 0.2, 0.2], 'BBOX_WEIGHTS': array([ 1., 1., 1., 1.]), 'BG_THRESH_HI': 0.5, 'BG_THRESH_LO': 0.0, 'CXX_PROPOSAL': True, 'END2END': False, 'FG_FRACTION': 0.25, 'FG_THRESH': 0.5, 'RPN_BATCH_SIZE': 256, 'RPN_BBOX_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'RPN_CLOBBER_POSITIVES': False, 'RPN_FG_FRACTION': 0.5, 'RPN_MIN_SIZE': 16, 'RPN_NEGATIVE_OVERLAP': 0.3, 'RPN_NMS_THRESH': 0.7, 'RPN_POSITIVE_OVERLAP': 0.7, 'RPN_POSITIVE_WEIGHT': -1.0, 'RPN_POST_NMS_TOP_N': 2000, 'RPN_PRE_NMS_TOP_N': 12000}} num_images 41 voc_2007_test gt roidb loaded from data/cache/voc_2007_test_gt_roidb.pkl /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/base_module.py:64: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['cls_prob_label']) warnings.warn(msg) [15:13:05] src/operator/./cudnn_convolution-inl.h:55: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) [15:13:11] /home/root1/new/software/mxnet/dmlc-core/include/dmlc/./logging.h:300: [15:13:11] src/operator/proposal.cu:496: Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
Stack trace returned 8 entries: [bt] (0) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f832ab6264c] [bt] (1) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2op13ProposalGPUOpIN7mshadow3gpuEE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESDSD+0x176e) [0x7f832c1e7d2e] [bt] (2) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf0dc9f) [0x7f832b41dc9f] [bt] (3) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x8c) [0x7f832b40913c] [bt] (4) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x60) [0x7f832b40c650] [bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f831dedba60] [bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f8338d10184] [bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8338a3d37d]
[15:13:11] /home/root1/new/software/mxnet/dmlc-core/include/dmlc/./logging.h:300: [15:13:11] src/engine/./threaded_engine.h:336: [15:13:11] src/operator/proposal.cu:496: Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
Stack trace returned 8 entries: [bt] (0) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f832ab6264c] [bt] (1) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2op13ProposalGPUOpIN7mshadow3gpuEE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESDSD+0x176e) [0x7f832c1e7d2e] [bt] (2) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf0dc9f) [0x7f832b41dc9f] [bt] (3) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x8c) [0x7f832b40913c] [bt] (4) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x60) [0x7f832b40c650] [bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f831dedba60] [bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f8338d10184] [bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8338a3d37d]
An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 6 entries: [bt] (0) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f832ab6264c] [bt] (1) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x371) [0x7f832b409421] [bt] (2) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x60) [0x7f832b40c650] [bt] (3) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f831dedba60] [bt] (4) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f8338d10184] [bt] (5) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8338a3d37d]
terminate called after throwing an instance of 'dmlc::Error' what(): [15:13:11] src/engine/./threaded_engine.h:336: [15:13:11] src/operator/proposal.cu:496: Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
Stack trace returned 8 entries: [bt] (0) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f832ab6264c] [bt] (1) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2op13ProposalGPUOpIN7mshadow3gpuEE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESDSD+0x176e) [0x7f832c1e7d2e] [bt] (2) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf0dc9f) [0x7f832b41dc9f] [bt] (3) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x8c) [0x7f832b40913c] [bt] (4) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x60) [0x7f832b40c650] [bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f831dedba60] [bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f8338d10184] [bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8338a3d37d]
An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 6 entries: [bt] (0) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f832ab6264c] [bt] (1) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x371) [0x7f832b409421] [bt] (2) /home/root1/.local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x60) [0x7f832b40c650] [bt] (3) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f831dedba60] [bt] (4) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f8338d10184] [bt] (5) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8338a3d37d] `
Minimum reproducible example
if you are using your own code, please provide a short script that reproduces the error.
Steps to reproduce
or if you are running standard examples, please provide the commands you have run that lead to the error.
What have you tried to solve it?
1.run the cmd : python test.py --gpu 0 (sometime it work) 2. 3.