search

hszhao / PSPNet

Pyramid Scene Parsing Network, CVPR2017.
https://hszhao.github.io/projects/pspnet
Other
1.58k stars 544 forks source link

evalution on cityscapes out of memory #71

Open yangia opened 6 years ago

yangia commented 6 years ago

I run the evaluation codes on cityscapes dataset and got an error of Check failed: error == cudaSuccess (2 vs. 0) out of memory. I have a GTX 1080 with 8G memory. I wonder if it was truly out of memory or I run into another unknown problem? Thank you. Here is the output `Cleared 0 solvers and 0 stand-alone nets processing 1 (1525)... W1211 09:01:54.356140 28282 net.hpp:42] DEPRECATED: ForwardPrefilled() will be removed in a future version. Use Forward(). F1211 09:01:54.816296 28282 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory Check failure stack trace: 

          abort() detected at Mon Dec 11 09:01:54 2017

Configuration: Crash Decoding : Disabled Crash Mode : continue (default) Current Graphics Driver: Unknown software Current Visual : None Default Encoding : UTF-8 GNU C Library : 2.23 stable Host Name : yang-ms-7885 MATLAB Architecture : glnxa64 MATLAB Root : /usr/local/MATLAB/R2015b MATLAB Version : 8.6.0.267246 (R2015b) OpenGL : software Operating System : Linux 4.4.0-101-generic #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 Processor ID : x86 Family 6 Model 79 Stepping 1, GenuineIntel Virtual Machine : Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode Window System : No active display

Fault Count: 1

Abnormal termination: abort()

Register State (from fault): RAX = 0000000000000000 RBX = 00007f16e8328420 RCX = 00007f1732ea4428 RDX = 0000000000000006 RSP = 00007f1717ff9018 RBP = 00007f1717ff92f0 RSI = 0000000000006e7a RDI = 0000000000006e44

R8 = 0000000000000081 R9 = 0000000000000000 R10 = 0000000000000008 R11 = 0000000000000206 R12 = 00007f16e8328480 R13 = 000000000000006a R14 = 00007f16e8328420 R15 = 00007f16e832fde0

RIP = 00007f1732ea4428 EFL = 0000000000000206

CS = 0033 FS = 0000 GS = 0000

Stack Trace (from fault): [ 0] 0x00007f1732ea4428 /lib/x86_64-linux-gnu/libc.so.6+00218152 gsignal+00000056 [ 1] 0x00007f1732ea602a /lib/x86_64-linux-gnu/libc.so.6+00225322 abort+00000362 [ 2] 0x00007f16e8113e49 /usr/lib/x86_64-linux-gnu/libglog.so.0+00040521 [ 3] 0x00007f16e81155cd /usr/lib/x86_64-linux-gnu/libglog.so.0+00046541 [ 4] 0x00007f16e8117433 /usr/lib/x86_64-linux-gnu/libglog.so.0+00054323 _ZN6google10LogMessage9SendToLogEv+00000643 [ 5] 0x00007f16e811515b /usr/lib/x86_64-linux-gnu/libglog.so.0+00045403 _ZN6google10LogMessage5FlushEv+00000187 [ 6] 0x00007f16e8117e1e /usr/lib/x86_64-linux-gnu/libglog.so.0+00056862 ZN6google15LogMessageFatalD2Ev+00000014 [ 7] 0x00007f168b5d9ec0 /home/yang/PSPNET-cudnn5/matlab/+caffe/private/caffe.mexa64+02850496 [ 8] 0x00007f168b5d8e89 /home/yang/PSPNET-cudnn5/matlab/+caffe/private/caffe.mexa64+02846345 [ 9] 0x00007f168b5e2172 /home/yang/PSPNET-cudnn5/matlab/+caffe/private/caffe.mexa64+02883954 [ 10] 0x00007f168b49e32f /home/yang/PSPNET-cudnn5/matlab/+caffe/private/caffe.mexa64+01557295 [ 11] 0x00007f168b436742 /home/yang/PSPNET-cudnn5/matlab/+caffe/private/caffe.mexa64+01132354 [ 12] 0x00007f168b436896 /home/yang/PSPNET-cudnn5/matlab/+caffe/private/caffe.mexa64+01132694 [ 13] 0x00007f168b371084 /home/yang/PSPNET-cudnn5/matlab/+caffe/private/caffe.mexa64+00323716 [ 14] 0x00007f168b372009 /home/yang/PSPNET-cudnn5/matlab/+caffe/private/caffe_.mexa64+00327689 mexFunction+00000169 [ 15] 0x00007f1725c98c4a /usr/local/MATLAB/R2015b/bin/glnxa64/libmex.so+00142410 mexRunMexFile+00000090 [ 16] 0x00007f1725c95244 /usr/local/MATLAB/R2015b/bin/glnxa64/libmex.so+00127556 [ 17] 0x00007f1725c95de4 /usr/local/MATLAB/R2015b/bin/glnxa64/libmex.so+00130532 [ 18] 0x00007f172a236dbd /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_dispatcher.so+00724413 _ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2iS2+00001549 [ 19] 0x00007f172a237250 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_dispatcher.so+00725584 _ZN8Mfh_file11dispatch_fhEiPP11mxArraytagiS2+00000032 [ 20] 0x00007f17248b28af /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+08612015 [ 21] 0x00007f17249d67ff /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09807871 [ 22] 0x00007f17249cc47f /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09766015 [ 23] 0x00007f1724999981 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09558401 [ 24] 0x00007f17245fbd6c /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05766508 [ 25] 0x00007f17245e64c1 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05678273 [ 26] 0x00007f17245f5075 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05738613 [ 27] 0x00007f17247c6033 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07643187 [ 28] 0x00007f17248a43bc /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+08553404 [ 29] 0x00007f172a236b31 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_dispatcher.so+00723761 _ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2iS2+00000897 [ 30] 0x00007f172a237250 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_dispatcher.so+00725584 _ZN8Mfh_file11dispatch_fhEiPP11mxArraytagiS2+00000032 [ 31] 0x00007f17248b28af /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+08612015 [ 32] 0x00007f17249d67ff /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09807871 [ 33] 0x00007f17249cc47f /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09766015 [ 34] 0x00007f1724999981 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09558401 [ 35] 0x00007f17245fbd6c /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05766508 [ 36] 0x00007f17245e64c1 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05678273 [ 37] 0x00007f17245f5075 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05738613 [ 38] 0x00007f17247c6033 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07643187 [ 39] 0x00007f172478dc40 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07412800 [ 40] 0x00007f1724790078 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07422072 [ 41] 0x00007f1724790140 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07422272 [ 42] 0x00007f17248076bc /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07911100 [ 43] 0x00007f1724807abc /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07912124 [ 44] 0x00007f1729767d0d /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_interpreter.so+02600205 _Z51inEvalCmdWithLocalReturnInDesiredWSAndPublishEventsRKSbIDsSt11char_traitsIDsESaIDsEEPibbP15inWorkSpace_tag+00000077 [ 45] 0x00007f172b362a12 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwiqm.so+00915986 _ZNK3iqm18InternalEvalPlugin24inEvalCmdWithLocalReturnERKSbIDsSt11char_traitsIDsESaIDsEEP15inWorkSpace_tag+00000098 [ 46] 0x00007f172b362bd8 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwiqm.so+00916440 _ZN3iqm18InternalEvalPlugin7executeEP15inWorkSpace_tagRN5boost10shared_ptrIN14cmddistributor17IIPCompletedEventEEE+00000120 [ 47] 0x00007f172a51b695 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00677525 [ 48] 0x00007f172b35c1c6 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwiqm.so+00889286 [ 49] 0x00007f172b349645 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwiqm.so+00812613 [ 50] 0x00007f1725ec6bf9 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwbridge.so+00146425 [ 51] 0x00007f1725ec71f4 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwbridge.so+00147956 [ 52] 0x00007f1725ecc6cd /usr/local/MATLAB/R2015b/bin/glnxa64/libmwbridge.so+00169677 [ 53] 0x00007f1725ecc7bc /usr/local/MATLAB/R2015b/bin/glnxa64/libmwbridge.so+00169916 [ 54] 0x00007f1725eccead /usr/local/MATLAB/R2015b/bin/glnxa64/libmwbridge.so+00171693 _Z8mnParserv+00000749 [ 55] 0x00007f172a51db4f /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00686927 _ZN11mcrInstance30mnParser_on_interpreter_threadEv+00000031 [ 56] 0x00007f172a50a443 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00607299 [ 57] 0x00007f172a50aa39 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00608825 _ZN5boost6detail11task_objectIvNS_3_bi6bind_tIvPFvRKNS_8functionIFvvEEEENS2_5list1INS2_5valueIS6_EEEEEEE6do_runEv+00000025 [ 58] 0x00007f172a50bf47 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00614215 _ZN5boost6detail9task_baseIvE3runEv+00000071 [ 59] 0x00007f172a50bfa7 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00614311 [ 60] 0x00007f172a5072fa /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00594682 [ 61] 0x00007f172ad9b7ab /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+01947563 [ 62] 0x00007f171e8126ed /usr/local/MATLAB/R2015b/bin/glnxa64/libmwuix.so+00206573 [ 63] 0x00007f172ae9d2ba /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+03003066 [ 64] 0x00007f172ae9d5f4 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+03003892 [ 65] 0x00007f172ae9ed9f /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+03009951 [ 66] 0x00007f172ae9f84c /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+03012684 _Z25svWS_ProcessPendingEventsiib+00000092 [ 67] 0x00007f172a5079b8 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00596408 [ 68] 0x00007f172a507cd4 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00597204 [ 69] 0x00007f172a4f3fed /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00516077 [ 70] 0x00007f17332406ba /lib/x86_64-linux-gnu/libpthread.so.0+00030394 [ 71] 0x00007f1732f763dd /lib/x86_64-linux-gnu/libc.so.6+01078237 clone+00000109 [ 72] 0x0000000000000000 +00000000

This error was detected while a MEX-file was running. If the MEX-file is not an official MathWorks function, please examine its source code for errors. Please consult the External Interfaces Guide for information on debugging MEX-files.

If this problem is reproducible, please submit a Service Request via: http://www.mathworks.com/support/contact_us/

A technical support engineer might contact you with further information.

Thank you for your help. This crash report has been saved to disk as /home/yang/matlab_crash_dump.28228-1

MATLAB is exiting because of fatal error Killed`

An-Pan commented 6 years ago

I encountered the same question

An-Pan commented 6 years ago

@yangia I can run this model on 1080ti after computer reboot. And from the ouput of "nvidia-smi" the card still free memory.