search

JackonYang / captcha-tensorflow

Image Captcha Solving Using TensorFlow and CNN Model. Accuracy 90%+
MIT License
996 stars 272 forks source link

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape #13

Closed aerosadegh closed 3 years ago

aerosadegh commented 6 years ago

#python cnn_1_char.py --data_dir=images/char-1-epoch-1000/

data loaded
train images: 10000. test images: 2000
label_size: 10, image_size: 6000
2018-03-18 15:52:23.154212: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-03-18 15:52:23.246932: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-18 15:52:23.247218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: GeForce GTX 1050 major: 6 minor: 1 memoryClockRate(GHz): 1.493
pciBusID: 0000:01:00.0
totalMemory: 3.95GiB freeMemory: 2.73GiB
2018-03-18 15:52:23.247233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-03-18 15:52:34.593156: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.43GiB.  Current allocation summary follows.
2018-03-18 15:52:34.593232: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256):   Total Chunks: 31, Chunks in use: 31. 7.8KiB allocated for chunks. 7.8KiB in use in bin. 2.1KiB client-requested in use in bin.
2018-03-18 15:52:34.593264: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512):   Total Chunks: 1, Chunks in use: 0. 768B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593289: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1024):  Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2018-03-18 15:52:34.593312: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2048):  Total Chunks: 5, Chunks in use: 5. 16.2KiB allocated for chunks. 16.2KiB in use in bin. 15.6KiB client-requested in use in bin.
2018-03-18 15:52:34.593333: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4096):  Total Chunks: 5, Chunks in use: 5. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin.
2018-03-18 15:52:34.593353: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8192):  Total Chunks: 1, Chunks in use: 1. 15.8KiB allocated for chunks. 15.8KiB in use in bin. 15.6KiB client-requested in use in bin.
2018-03-18 15:52:34.593373: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16384):         Total Chunks: 1, Chunks in use: 0. 21.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593394: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (32768):         Total Chunks: 4, Chunks in use: 4. 160.0KiB allocated for chunks. 160.0KiB in use in bin. 160.0KiB client-requested in use in bin.
2018-03-18 15:52:34.593415: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (65536):         Total Chunks: 2, Chunks in use: 1. 160.0KiB allocated for chunks. 78.2KiB in use in bin. 78.1KiB client-requested in use in bin.
2018-03-18 15:52:34.593436: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (131072):        Total Chunks: 4, Chunks in use: 4. 800.0KiB allocated for chunks. 800.0KiB in use in bin. 800.0KiB client-requested in use in bin.
2018-03-18 15:52:34.593453: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (262144):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593471: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (524288):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593488: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1048576):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593505: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2097152):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593522: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4194304):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593539: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8388608):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593557: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593578: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (33554432):      Total Chunks: 2, Chunks in use: 1. 93.75MiB allocated for chunks. 45.78MiB in use in bin. 45.78MiB client-requested in use in bin.
2018-03-18 15:52:34.593599: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (67108864):      Total Chunks: 4, Chunks in use: 4. 375.00MiB allocated for chunks. 375.00MiB in use in bin. 375.00MiB client-requested in use in bin.
2018-03-18 15:52:34.593616: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-03-18 15:52:34.593636: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 1.98GiB allocated for chunks. 1.98GiB in use in bin. 1.43GiB client-requested in use in bin.
2018-03-18 15:52:34.593654: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin for 1.43GiB was 256.00MiB, Chunk State: 
2018-03-18 15:52:34.593673: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c00000 of size 1280
2018-03-18 15:52:34.593688: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c00500 of size 256
2018-03-18 15:52:34.593701: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c00600 of size 256
2018-03-18 15:52:34.593715: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c00700 of size 256
2018-03-18 15:52:34.593728: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c00800 of size 256
2018-03-18 15:52:34.593743: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c00900 of size 40960
2018-03-18 15:52:34.593757: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c0a900 of size 4096
2018-03-18 15:52:34.593770: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c0b900 of size 256
2018-03-18 15:52:34.593784: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c0ba00 of size 256
2018-03-18 15:52:34.593797: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c0bb00 of size 256
2018-03-18 15:52:34.593811: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c0bc00 of size 3328
2018-03-18 15:52:34.593824: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c0c900 of size 256
2018-03-18 15:52:34.593843: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c0ca00 of size 204800
2018-03-18 15:52:34.593857: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c3ea00 of size 256
2018-03-18 15:52:34.593870: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c3eb00 of size 256
2018-03-18 15:52:34.593884: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0513c3ec00 of size 98304000
2018-03-18 15:52:34.593898: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f05199fec00 of size 4096
2018-03-18 15:52:34.593911: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f05199ffc00 of size 256
2018-03-18 15:52:34.593924: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f05199ffd00 of size 256
2018-03-18 15:52:34.593937: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f05199ffe00 of size 256
2018-03-18 15:52:34.593951: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f05199fff00 of size 256
2018-03-18 15:52:34.593964: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a00000 of size 256
2018-03-18 15:52:34.593977: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a00100 of size 256
2018-03-18 15:52:34.593990: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a00200 of size 256
2018-03-18 15:52:34.594003: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a00300 of size 256
2018-03-18 15:52:34.594016: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a00400 of size 256
2018-03-18 15:52:34.594029: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a00500 of size 256
2018-03-18 15:52:34.594043: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a00900 of size 256
2018-03-18 15:52:34.594056: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a00a00 of size 40960
2018-03-18 15:52:34.594070: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a0aa00 of size 80128
2018-03-18 15:52:34.594084: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a32a00 of size 256
2018-03-18 15:52:34.594097: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0519a32b00 of size 48000000
2018-03-18 15:52:34.594111: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f7f2b00 of size 40960
2018-03-18 15:52:34.594123: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f7fcb00 of size 4096
2018-03-18 15:52:34.594137: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f7fdb00 of size 16128
2018-03-18 15:52:34.594150: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f801a00 of size 3328
2018-03-18 15:52:34.594164: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f807b00 of size 40960
2018-03-18 15:52:34.594178: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f811b00 of size 256
2018-03-18 15:52:34.594191: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f811c00 of size 256
2018-03-18 15:52:34.594204: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f811d00 of size 256
2018-03-18 15:52:34.594218: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f811e00 of size 3328
2018-03-18 15:52:34.594231: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f812b00 of size 3328
2018-03-18 15:52:34.594244: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f813800 of size 256
2018-03-18 15:52:34.594258: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f813900 of size 256
2018-03-18 15:52:34.594271: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f813a00 of size 204800
2018-03-18 15:52:34.594284: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f845a00 of size 204800
2018-03-18 15:52:34.594298: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f877a00 of size 256
2018-03-18 15:52:34.594311: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f877b00 of size 256
2018-03-18 15:52:34.594324: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f877c00 of size 256
2018-03-18 15:52:34.594338: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f051f877d00 of size 98304000
2018-03-18 15:52:34.594351: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f0525637d00 of size 98304000
2018-03-18 15:52:34.594364: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f052b3f7d00 of size 256
2018-03-18 15:52:34.594377: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f052b3f7e00 of size 4096
2018-03-18 15:52:34.594391: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f052b3f8e00 of size 4096
2018-03-18 15:52:34.594404: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f052b3f9e00 of size 3328
2018-03-18 15:52:34.594417: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f052b3fab00 of size 204800
2018-03-18 15:52:34.594430: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f052b42cb00 of size 98304000
2018-03-18 15:52:34.594444: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f05311ecb00 of size 2126722304
2018-03-18 15:52:34.594458: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f0519a00600 of size 768
2018-03-18 15:52:34.594472: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f0519a1e300 of size 83712
2018-03-18 15:52:34.594485: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f051c7f9700 of size 50304000
2018-03-18 15:52:34.594499: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f051f802700 of size 21504
2018-03-18 15:52:34.594512: I tensorflow/core/common_runtime/bfc_allocator.cc:676]      Summary of in-use Chunks by size: 
2018-03-18 15:52:34.594529: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 31 Chunks of size 256 totalling 7.8KiB
2018-03-18 15:52:34.594545: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1280 totalling 1.2KiB
2018-03-18 15:52:34.594561: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 5 Chunks of size 3328 totalling 16.2KiB
2018-03-18 15:52:34.594577: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 5 Chunks of size 4096 totalling 20.0KiB
2018-03-18 15:52:34.594593: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 16128 totalling 15.8KiB
2018-03-18 15:52:34.594608: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 40960 totalling 160.0KiB
2018-03-18 15:52:34.594624: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 80128 totalling 78.2KiB
2018-03-18 15:52:34.594639: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 204800 totalling 800.0KiB
2018-03-18 15:52:34.594655: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 48000000 totalling 45.78MiB
2018-03-18 15:52:34.594671: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 98304000 totalling 375.00MiB
2018-03-18 15:52:34.594686: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 2126722304 totalling 1.98GiB
2018-03-18 15:52:34.594701: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 2.39GiB
2018-03-18 15:52:34.594721: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats: 
Limit:                  2619473920
InUse:                  2569063936
MaxInUse:               2569063936
NumAllocs:                     189
MaxAllocSize:           2126722304

2018-03-18 15:52:34.594755: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ******_***********************************************************************xxxxxxxxxxxxxxxxxxxxxx
2018-03-18 15:52:34.594788: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[2000,32,100,60]
Traceback (most recent call last):
  File "cnn_1_char.py", line 167, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "cnn_1_char.py", line 148, in main
    test_summary, test_accuracy = sess.run([merged, accuracy], feed_dict={x: test_x, y_: test_y, keep_prob: 1.0})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2000,32,100,60]
         [[Node: convolution-layer-1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](input/Reshape, convolution-layer-1/Variable/read)]]

Caused by op u'convolution-layer-1/Conv2D', defined at:
  File "cnn_1_char.py", line 167, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "cnn_1_char.py", line 75, in main
    h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
  File "cnn_1_char.py", line 41, in conv2d
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 631, in conv2d
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[2000,32,100,60]
         [[Node: convolution-layer-1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](input/Reshape, convolution-layer-1/Variable/read)]]

How can I solve it?

JackonYang commented 6 years ago

reduce the batch size as you have only 4G GPU memory.

aerosadegh commented 6 years ago

Thank you! I got it! How can I save the trained model?

HytonightYX commented 6 years ago

Mr.Yang, My computer only has MX-150, 2GB, and I want to use cnn_n_char.py, even I changed batch size to 1, I also met this porblem.Does this mean I can t to use my computer to train?