search

apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.73k stars 6.81k forks source link

When I run ssd example on cpu computer, I get "Check failed: temp.size() >= num_negative (0 vs. 3) error" error. #5431

Closed aaronshan closed 6 years ago

aaronshan commented 7 years ago

Environment info

Operating System: centos 7

$ cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
24  Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz

Compiler: gcc 4.8.5

Package used (Python/R/Scala/Julia): python 2.7.12

MXNet version:

Or if installed from source: yes, 0.8.0

MXNet commit hash (git rev-parse HEAD): 8b28b8c159a941123731fbbd9dcd06bfc1af3556

If you are using python package, please provide

Python version and distribution: python 2.7.12

If you are using R package, please provide

R sessionInfo():

Error Message:

Please paste the full error message, including stack trace.

[ruifeng.shan@data.cn1 /home/q/mxnet/example/ssd]$ python train.py
INFO:root:Start training with (cpu(0)) from pretrained model /home/q/mxnet/example/ssd/model/vgg16_reduced
[19:50:06] src/nnvm/legacy_json_util.cc:175: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
INFO:root:Freezed parameters: [conv1_1_weight,conv1_1_bias,conv1_2_weight,conv1_2_bias,conv2_1_weight,conv2_1_bias,conv2_2_weight,conv2_2_bias]
[19:51:04] /home/q/mxnet/dmlc-core/include/dmlc/./logging.h:300: [19:51:04] example/ssd/operator/multibox_target.cc:215: Check failed: temp.size() >= num_negative (0 vs. 3)

Stack trace returned 9 entries:
[bt] (0) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x29) [0x7fcba685ab79]
[bt] (1) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN7mshadow21MultiBoxTargetForwardIfEEvRKNS_6TensorINS_3cpuELi2ET_EES6_S6_S6_RKNS1_IS2_Li3ES3_EES9_RKNS1_IS2_Li4ES3_EEffffiRKSt6vectorIfSaIfEE+0x1da1) [0x7fcba745aa31]
[bt] (2) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2op16MultiBoxTargetOpIN7mshadow3cpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD_+0x1dba) [0x7fcba7464e1a]
[bt] (3) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xe8694f) [0x7fcba740e94f]
[bt] (4) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x7f) [0x7fcba73cafff]
[bt] (5) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x2e) [0x7fcba73cd84e]
[bt] (6) /home/q/anaconda/anaconda2/bin/../lib/libstdc++.so.6(+0xb4870) [0x7fcba27cb870]
[bt] (7) /lib64/libpthread.so.0(+0x7dc5) [0x7fcbb74d5dc5]
[bt] (8) /lib64/libc.so.6(clone+0x6d) [0x7fcbb6af9ced]

[19:51:04] /home/q/mxnet/dmlc-core/include/dmlc/./logging.h:300: [19:51:04] src/engine/./threaded_engine.h:336: [19:51:04] example/ssd/operator/multibox_target.cc:215: Check failed: temp.size() >= num_negative (0 vs. 3)

Stack trace returned 9 entries:
[bt] (0) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x29) [0x7fcba685ab79]
[bt] (1) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN7mshadow21MultiBoxTargetForwardIfEEvRKNS_6TensorINS_3cpuELi2ET_EES6_S6_S6_RKNS1_IS2_Li3ES3_EES9_RKNS1_IS2_Li4ES3_EEffffiRKSt6vectorIfSaIfEE+0x1da1) [0x7fcba745aa31]
[bt] (2) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2op16MultiBoxTargetOpIN7mshadow3cpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD_+0x1dba) [0x7fcba7464e1a]
[bt] (3) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xe8694f) [0x7fcba740e94f]
[bt] (4) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x7f) [0x7fcba73cafff]
[bt] (5) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x2e) [0x7fcba73cd84e]
[bt] (6) /home/q/anaconda/anaconda2/bin/../lib/libstdc++.so.6(+0xb4870) [0x7fcba27cb870]
[bt] (7) /lib64/libpthread.so.0(+0x7dc5) [0x7fcbb74d5dc5]
[bt] (8) /lib64/libc.so.6(clone+0x6d) [0x7fcbb6af9ced]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 6 entries:
[bt] (0) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x29) [0x7fcba685ab79]
[bt] (1) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x301) [0x7fcba73cb281]
[bt] (2) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x2e) [0x7fcba73cd84e]
[bt] (3) /home/q/anaconda/anaconda2/bin/../lib/libstdc++.so.6(+0xb4870) [0x7fcba27cb870]
[bt] (4) /lib64/libpthread.so.0(+0x7dc5) [0x7fcbb74d5dc5]
[bt] (5) /lib64/libc.so.6(clone+0x6d) [0x7fcbb6af9ced]

terminate called after throwing an instance of 'dmlc::Error'
  what():  [19:51:04] src/engine/./threaded_engine.h:336: [19:51:04] example/ssd/operator/multibox_target.cc:215: Check failed: temp.size() >= num_negative (0 vs. 3)

Stack trace returned 9 entries:
[bt] (0) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x29) [0x7fcba685ab79]
[bt] (1) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN7mshadow21MultiBoxTargetForwardIfEEvRKNS_6TensorINS_3cpuELi2ET_EES6_S6_S6_RKNS1_IS2_Li3ES3_EES9_RKNS1_IS2_Li4ES3_EEffffiRKSt6vectorIfSaIfEE+0x1da1) [0x7fcba745aa31]
[bt] (2) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2op16MultiBoxTargetOpIN7mshadow3cpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD_+0x1dba) [0x7fcba7464e1a]
[bt] (3) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xe8694f) [0x7fcba740e94f]
[bt] (4) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x7f) [0x7fcba73cafff]
[bt] (5) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x2e) [0x7fcba73cd84e]
[bt] (6) /home/q/anaconda/anaconda2/bin/../lib/libstdc++.so.6(+0xb4870) [0x7fcba27cb870]
[bt] (7) /lib64/libpthread.so.0(+0x7dc5) [0x7fcbb74d5dc5]
[bt] (8) /lib64/libc.so.6(clone+0x6d) [0x7fcbb6af9ced]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 6 entries:
[bt] (0) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x29) [0x7fcba685ab79]
[bt] (1) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x301) [0x7fcba73cb281]
[bt] (2) /home/q/anaconda/anaconda2/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x2e) [0x7fcba73cd84e]
[bt] (3) /home/q/anaconda/anaconda2/bin/../lib/libstdc++.so.6(+0xb4870) [0x7fcba27cb870]
[bt] (4) /lib64/libpthread.so.0(+0x7dc5) [0x7fcbb74d5dc5]
[bt] (5) /lib64/libc.so.6(clone+0x6d) [0x7fcbb6af9ced]

Aborted

Minimum reproducible example

if you are using your own code, please provide a short script that reproduces the error.

Steps to reproduce

or if you are running standard examples, please provide the commands you have run that lead to the error.

  1. install mxnet by offical instruction.
  2. download dataset by ssd readme text.
  3. cd /home/q/mxnet/example/ssd and run python train.py

What have you tried to solve it?

I searched by google, and someone faced this error also. but nobody share their solutions.

I'm a newer for deep learn and mxnet. only have cpu machine, maybe I should buy a computer with N card.

Thanks for your reply.

ikvision commented 7 years ago

I got the same error on CPU os X python 3.5 Any update on this issue?

yajiedesign commented 6 years ago

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!