search

zhreshold / mxnet-yolo

YOLO: You only look once real-time object detector
MIT License
239 stars 88 forks source link

Can not run using GPU #28

Closed nvnnghia closed 6 years ago

nvnnghia commented 6 years ago

Hello, thank you for sharing the code. I am tried to run your code using gpu. I met the error. Please help me. I run with: python demo.py --gpu Error is: 

[09:41:08] src/operator/././cudnn_algoreg-inl.h:65: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[09:41:26] /home/nvidia/mxnet-yolo/mxnet/dmlc-core/include/dmlc/./logging.h:304: [09:41:26] src/operator/./cudnn_convolution-inl.h:583: Check failed: e == CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR

Stack trace returned 8 entries:
[bt] (0) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x44) [0x7fa6f6d38c]
[bt] (1) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZZN5mxnet2op18CuDNNConvolutionOpIfE10SelectAlgoERKNS_7ContextERKSt6vectorIN4nnvm6TShapeESaIS8_EESC_15cudnnDataType_tSD_ENKUlNS_10RunContextEE_clESE_+0x484) [0x7fa85d05d4]
[bt] (2) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvN5mxnet10RunContextENS0_6engine18CallbackOnCompleteEEZNS0_6Engine8PushSyncESt8functionIFvS1_EENS0_7ContextERKSt6vectorIPNS2_3VarESaISC_EESG_NS0_10FnPropertyEiPKcEUlS1_S3_E_E9_M_invokeERKSt9_Any_dataOS1_OS3_+0x4c) [0x7fa77a025c]
[bt] (3) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7fa7b2c508]
[bt] (4) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5_+0x110) [0x7fa7b37518]
[bt] (5) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x48) [0x7fa7b2eb50]
[bt] (6) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f8b15d280]
[bt] (7) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7fb3596fc4]

[09:41:26] /home/nvidia/mxnet-yolo/mxnet/dmlc-core/include/dmlc/./logging.h:304: [09:41:26] src/engine/./threaded_engine.h:329: [09:41:26] src/operator/./cudnn_convolution-inl.h:583: Check failed: e == CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR

Stack trace returned 8 entries:
[bt] (0) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x44) [0x7fa6f6d38c]
[bt] (1) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZZN5mxnet2op18CuDNNConvolutionOpIfE10SelectAlgoERKNS_7ContextERKSt6vectorIN4nnvm6TShapeESaIS8_EESC_15cudnnDataType_tSD_ENKUlNS_10RunContextEE_clESE_+0x484) [0x7fa85d05d4]
[bt] (2) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvN5mxnet10RunContextENS0_6engine18CallbackOnCompleteEEZNS0_6Engine8PushSyncESt8functionIFvS1_EENS0_7ContextERKSt6vectorIPNS2_3VarESaISC_EESG_NS0_10FnPropertyEiPKcEUlS1_S3_E_E9_M_invokeERKSt9_Any_dataOS1_OS3_+0x4c) [0x7fa77a025c]
[bt] (3) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7fa7b2c508]
[bt] (4) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5_+0x110) [0x7fa7b37518]
[bt] (5) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x48) [0x7fa7b2eb50]
[bt] (6) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f8b15d280]
[bt] (7) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7fb3596fc4]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 6 entries:
[bt] (0) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x44) [0x7fa6f6d38c]
[bt] (1) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2c4) [0x7fa7b2c744]
[bt] (2) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5_+0x110) [0x7fa7b37518]
[bt] (3) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x48) [0x7fa7b2eb50]
[bt] (4) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f8b15d280]
[bt] (5) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7fb3596fc4]

terminate called after throwing an instance of 'dmlc::Error'
  what():  [09:41:26] src/engine/./threaded_engine.h:329: [09:41:26] src/operator/./cudnn_convolution-inl.h:583: Check failed: e == CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR

Stack trace returned 8 entries:
[bt] (0) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x44) [0x7fa6f6d38c]
[bt] (1) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZZN5mxnet2op18CuDNNConvolutionOpIfE10SelectAlgoERKNS_7ContextERKSt6vectorIN4nnvm6TShapeESaIS8_EESC_15cudnnDataType_tSD_ENKUlNS_10RunContextEE_clESE_+0x484) [0x7fa85d05d4]
[bt] (2) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvN5mxnet10RunContextENS0_6engine18CallbackOnCompleteEEZNS0_6Engine8PushSyncESt8functionIFvS1_EENS0_7ContextERKSt6vectorIPNS2_3VarESaISC_EESG_NS0_10FnPropertyEiPKcEUlS1_S3_E_E9_M_invokeERKSt9_Any_dataOS1_OS3_+0x4c) [0x7fa77a025c]
[bt] (3) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7fa7b2c508]
[bt] (4) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5_+0x110) [0x7fa7b37518]
[bt] (5) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x48) [0x7fa7b2eb50]
[bt] (6) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f8b15d280]
[bt] (7) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7fb3596fc4]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 6 entries:
[bt] (0) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x44) [0x7fa6f6d38c]
[bt] (1) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2c4) [0x7fa7b2c744]
[bt] (2) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5_+0x110) [0x7fa7b37518]
[bt] (3) /home/nvidia/mxnet-yolo/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x48) [0x7fa7b2eb50]
[bt] (4) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f8b15d280]
[bt] (5) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7fb3596fc4]

Aborted (core dumped)

Thank you very much

zhreshold commented 6 years ago

CUDNN_STATUS_INTERNAL_ERROR suggest some problem with cudnn, I don't have much experience with that.

nvnnghia commented 6 years ago

Thank you for replying