search

itijyou / ademxapp

Code for https://arxiv.org/abs/1611.10080
Other
340 stars 98 forks source link

Errors in full image test on cityscapes val dataset #32

Closed yzou2 closed 7 years ago

yzou2 commented 7 years ago

I use the following codes for segmentation cityscapes val dataset.

python issegm/voc.py --data-root data/cityscapes --output output --phase val --weights models/cityscapes_rna-a1_cls19_s8_ep-0001.params --split val --test-scales 2048 --test-flipping --gpus 0

But I get the errors that are given below. However, if decrease the --test-scales to 1800, everything runs smoothly without troubles? I am sure it's not because of the GPU memory issues (I use a Titan X 12 Gb GPU). Any hints why this happens?

[22:28:43] /home/travis/build/dmlc/mxnet-distro/mxnet-build/dmlc-core/include/dmlc/logging.h:304: [22:28:43] src/operator/./cudnn_convolution-inl.h:572: Check failed: e == CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR

Stack trace returned 8 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x18b0dc) [0x7f5dd48610dc] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x1a64e8f) [0x7f5dd613ae8f] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x21e123) [0x7f5dd48f4123] [bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb7e5bc) [0x7f5dd52545bc] [bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb81590) [0x7f5dd5257590] [bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f5de7e91c80] [bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f5dee6566fa] [bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5dee38cb5d]

[22:28:43] /home/travis/build/dmlc/mxnet-distro/mxnet-build/dmlc-core/include/dmlc/logging.h:304: [22:28:43] src/engine/./threaded_engine.h:329: [22:28:43] src/operator/./cudnn_convolution-inl.h:572: Check failed: e == CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR

Stack trace returned 8 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x18b0dc) [0x7f5dd48610dc] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x1a64e8f) [0x7f5dd613ae8f] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x21e123) [0x7f5dd48f4123] [bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb7e5bc) [0x7f5dd52545bc] [bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb81590) [0x7f5dd5257590] [bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f5de7e91c80] [bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f5dee6566fa] [bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5dee38cb5d]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 6 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x18b0dc) [0x7f5dd48610dc] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb7e84f) [0x7f5dd525484f] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb81590) [0x7f5dd5257590] [bt] (3) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f5de7e91c80] [bt] (4) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f5dee6566fa] [bt] (5) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5dee38cb5d]

terminate called after throwing an instance of 'dmlc::Error' what(): [22:28:43] src/engine/./threaded_engine.h:329: [22:28:43] src/operator/./cudnn_convolution-inl.h:572: Check failed: e == CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR

Stack trace returned 8 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x18b0dc) [0x7f5dd48610dc] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x1a64e8f) [0x7f5dd613ae8f] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x21e123) [0x7f5dd48f4123] [bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb7e5bc) [0x7f5dd52545bc] [bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb81590) [0x7f5dd5257590] [bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f5de7e91c80] [bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f5dee6566fa] [bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5dee38cb5d]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 6 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x18b0dc) [0x7f5dd48610dc] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb7e84f) [0x7f5dd525484f] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xb81590) [0x7f5dd5257590] [bt] (3) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f5de7e91c80] [bt] (4) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f5dee6566fa] [bt] (5) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5dee38cb5d]

yzou2 commented 7 years ago

I solve this problem by disabling cudnn.

eveningdong commented 7 years ago

Hi, I know this is closed. Just want to help the future readers if they are new to MXNET like me. MXNET has problem with cuDNN when the image is large, one simple example is Cityscapes. Anyone meet this problem should export MXNET_CUDNN_AUTOTUNE_DEFAULT=0.