search

hpi-xnor / BMXNet

(New version is out: https://github.com/hpi-xnor/BMXNet-v2) BMXNet: An Open-Source Binary Neural Network Implementation Based on MXNet
Apache License 2.0
349 stars 95 forks source link

CUDA: an illegal memory access was encountered when training on GPU #14

Closed luxial closed 6 years ago

luxial commented 6 years ago

I want to train binary/8bit on imagenet using train_imagenet.py under BMXNet/smd_hpi/examples/binary-imagenet1k. I did a few line of change inside parser.set_defaults(), as following:

parser.set_defaults(

network

    network        = 'resnet-binary',  //changed from 'resnet'
    num_layers     = 18,

    # data
    num_classes      = 1000,
    num_examples     = 1281167,
    image_shape      = '3,224,224',
    min_random_scale = 1, # if input image has min size k, suggest to use
                          # 256.0/x, e.g. 0.533 for 480
    # train
    num_epochs       = 60,
    lr_step_epochs   = '20,40',
    lr               = 0.001,
    lr_factor        = 0.1,
    batch_size     = 32,
    optimizer        = 'sgd',
    disp_batches     = 10,
    top_k            = 5,
    act_bit          = 8, //added this line, or act_bit=1 or 32 all not working on GPU
)

I used this command to train: python train_imagenet.py --gpus '0' --data-train imagenet-train.rec --data-val imagenet-val.rec --batch-size 32

If I use GPU, it gives a CUDA illegal memory access error, if training on CPU (--gpu '') the training proceed fine. Please suggest the possible problem. I tried on two different ubuntu machine, the same thing. Thank you!

Environment info

Ubuntu 16.04 cuda-8.0 cudnn6 Python 2.7.12 GTX 1080 Ti Lastest BMXNet code at commit c6624e

Error Message:

INFO:root:start with arguments Namespace(act_bit=8, batch_size=32, benchmark=0, data_nthreads=4, data_train='/home/local/ANT/luxial/DATA/ImageNet/imagenet-train.rec', data_val='/home/local/ANT/luxial/DATA/ImageNet/imagenet-val.rec', disp_batches=10, gpus='0', image_shape='3,224,224', kv_store='device', load_epoch=None, log_file='train.log', lr=0.001, lr_factor=0.1, lr_step_epochs='20,40', max_random_aspect_ratio=0.25, max_random_h=36, max_random_l=50, max_random_rotate_angle=10, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0.1, min_random_scale=1, model_prefix='./myImageNetModels/resnet-18-8bit', mom=0.9, monitor=0, network='resnet-binary', num_classes=1000, num_epochs=60, num_examples=1281167, num_layers=18, optimizer='sgd', pad_size=0, pretrained=None, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=5, wd=0.0001) [19:47:51] /home/local/ANT/luxial/BMXNet/src/io/iter_image_recordio_2.cc:135: ImageRecordIOParser2: /home/local/ANT/luxial/DATA/ImageNet/imagenet-train.rec, use 3 threads for decoding.. [19:47:53] /home/local/ANT/luxial/BMXNet/src/io/iter_image_recordio_2.cc:135: ImageRecordIOParser2: /home/local/ANT/luxial/DATA/ImageNet/imagenet-val.rec, use 3 threads for decoding.. [19:47:54] /home/local/ANT/luxial/BMXNet/src/operator/././cudnn_algoreg-inl.h:65: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) [19:47:55] /home/local/ANT/luxial/BMXNet/dmlc-core/include/dmlc/./logging.h:308: [19:47:55] /home/local/ANT/luxial/BMXNet/mshadow/mshadow/./stream_gpu-inl.h:55: Check failed: e == cudaSuccess CUDA: an illegal memory access was encountered

Stack trace returned 9 entries: [bt] (0) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x39) [0x7f236fa98cd9] [bt] (1) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN7mshadow6StreamINS_3gpuEE4WaitEv+0xd8) [0x7f236fabef98] [bt] (2) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(+0x6b39af) [0x7f236faa59af] [bt] (3) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x93) [0x7f2370a8bce3] [bt] (4) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_AnydataOS5+0x123) [0x7f2370a95693] [bt] (5) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x4e) [0x7f2370a8e3ee] [bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f234b12cc80] [bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f239036c6ba] [bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f23900a23dd]

[19:47:55] /home/local/ANT/luxial/BMXNet/dmlc-core/include/dmlc/./logging.h:308: [19:47:55] /home/local/ANT/luxial/BMXNet/src/engine/./threaded_engine.h:329: [19:47:55] /home/local/ANT/luxial/BMXNet/mshadow/mshadow/./stream_gpu-inl.h:55: Check failed: e == cudaSuccess CUDA: an illegal memory access was encountered

Stack trace returned 9 entries: [bt] (0) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x39) [0x7f236fa98cd9] [bt] (1) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN7mshadow6StreamINS_3gpuEE4WaitEv+0xd8) [0x7f236fabef98] [bt] (2) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(+0x6b39af) [0x7f236faa59af] [bt] (3) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x93) [0x7f2370a8bce3] [bt] (4) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_AnydataOS5+0x123) [0x7f2370a95693] [bt] (5) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x4e) [0x7f2370a8e3ee] [bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f234b12cc80] [bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f239036c6ba] [bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f23900a23dd]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 7 entries: [bt] (0) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x39) [0x7f236fa98cd9] [bt] (1) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x36b) [0x7f2370a8bfbb] [bt] (2) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_AnydataOS5+0x123) [0x7f2370a95693] [bt] (3) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x4e) [0x7f2370a8e3ee] [bt] (4) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f234b12cc80] [bt] (5) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f239036c6ba] [bt] (6) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f23900a23dd]

terminate called after throwing an instance of 'dmlc::Error' what(): [19:47:55] /home/local/ANT/luxial/BMXNet/src/engine/./threaded_engine.h:329: [19:47:55] /home/local/ANT/luxial/BMXNet/mshadow/mshadow/./stream_gpu-inl.h:55: Check failed: e == cudaSuccess CUDA: an illegal memory access was encountered

Stack trace returned 9 entries: [bt] (0) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x39) [0x7f236fa98cd9] [bt] (1) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN7mshadow6StreamINS_3gpuEE4WaitEv+0xd8) [0x7f236fabef98] [bt] (2) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(+0x6b39af) [0x7f236faa59af] [bt] (3) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x93) [0x7f2370a8bce3] [bt] (4) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_AnydataOS5+0x123) [0x7f2370a95693] [bt] (5) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x4e) [0x7f2370a8e3ee] [bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f234b12cc80] [bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f239036c6ba] [bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f23900a23dd]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 7 entries: [bt] (0) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x39) [0x7f236fa98cd9] [bt] (1) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x36b) [0x7f2370a8bfbb] [bt] (2) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_AnydataOS5+0x123) [0x7f2370a95693] [bt] (3) /home/local/ANT/luxial/BMXNet/build/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x4e) [0x7f2370a8e3ee] [bt] (4) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f234b12cc80] [bt] (5) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f239036c6ba] [bt] (6) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f23900a23dd]

P.S.

  1. I am able to train standard network on GPU with the same script (e.g. network = 'resnet'). Just not the quantized version where QActivation is used.
  2. Evaluation of the pretrained network BMXNet/smd_hpi/binary_models/binarized_imagenet-resnet18-64bit or binarized_imagenet-resnet18-64bit-1st-stage-fullprecision is the same thing, only works for CPU.
yanghaojin commented 6 years ago

Hi, we will investigate this issue, thank you for your report. There might be bugs in the new "q_convolution" layer. You could also try to fallback to old version "q_convolution_v1" to check out.

Note that to evaluate the pre-tained binarized models, you have to use the CPU modus. In our evaluation, we found that implementing a conv-layer by using binary operator xnor and popc in CUDA is far slower than cudnn-Conv. And currently, the most urgently needed use cases are almost from the CPU domain such as mobile devices, embedded devices, on which we cannot install a Nvidia GPU... Therefore the idea of this implementation is to enable binary model training using GPU (after the training the binary weights are still saving in float), we can subsequently binarize the trained model by using model converter (concatenate weights into binary model), then apply the binarized model using CPU. We priveded 2 image classification examples for both ios and android system using binarized model. Thus in BMXNet the binary convolution/fc are implemented using CPU instruction such as xnor and popc .

mtin commented 6 years ago

please try again after updating, there was a bug in our memory management. report back if you still have any trouble!

luxial commented 6 years ago

Thank you for the prompt fix! The training is able to proceed on GPU now. However, I noticed training 'resnet-binary' with act_bit = 1 is about 10x slower than standard 'resnet': resnet-binary: INFO:root:Epoch[0] Batch [10] Speed: 32.09 samples/sec accuracy=0.001420 top_k_accuracy_5=0.006392 INFO:root:Epoch[0] Batch [20] Speed: 32.03 samples/sec accuracy=0.000781 top_k_accuracy_5=0.007812 INFO:root:Epoch[0] Batch [30] Speed: 32.02 samples/sec accuracy=0.000781 top_k_accuracy_5=0.003906 INFO:root:Epoch[0] Batch [40] Speed: 32.02 samples/sec accuracy=0.001563 top_k_accuracy_5=0.004687

resnet: INFO:root:Epoch[0] Batch [10] Speed: 358.22 samples/sec accuracy=0.000710 top_k_accuracy_5=0.005682 INFO:root:Epoch[0] Batch [20] Speed: 358.64 samples/sec accuracy=0.000781 top_k_accuracy_5=0.004687 INFO:root:Epoch[0] Batch [30] Speed: 357.47 samples/sec accuracy=0.000781 top_k_accuracy_5=0.000781 INFO:root:Epoch[0] Batch [40] Speed: 357.09 samples/sec accuracy=0.000781 top_k_accuracy_5=0.002344

I am using the batch_size = 128 with 1 GPU (1080Ti) on the same machine. I understand that since "resnet-binary" is doing the conversion and saving values with floating point still during training, it can be slower. But is this 10x expected? Thanks.