search

faustomorales / keras-ocr

A packaged and flexible version of the CRAFT text detector and Keras CRNN recognition model.
https://keras-ocr.readthedocs.io/
MIT License
1.37k stars 349 forks source link

RESOURCE_EXHAUSTED: failed to allocate memory #195

Open AIMLAPP opened 2 years ago

AIMLAPP commented 2 years ago

Hi,

I am currently running the code detector = keras_ocr.detection.Detector(weights='clovaai_general') on a Tesla K80 GPU with 4v CPUs and 15GB RAM. However, I get an error (please refer to end of this message).

I have run the code assert tf.test.is_gpu_available() to check for my GPU and obtain the following:

2022-02-11 13:07:24.961634: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-11 13:07:24.962101: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-11 13:07:24.962404: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-11 13:07:24.963086: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-11 13:07:24.963391: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-11 13:07:24.963804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /device:GPU:0 with 203 MB memory:  -> device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7

Would anyone be able to help please? Thank you very much.

Error Message

Looking for /home/jupyter/.keras-ocr/craft_mlt_25k.h5
2022-02-11 12:56:32.748705: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 20.34M (21331968 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2022-02-11 12:56:32.759614: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 20.34M (21331968 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2022-02-11 12:56:42.770253: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 20.34M (21331968 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2022-02-11 12:56:42.781185: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 20.34M (21331968 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2022-02-11 12:56:42.781230: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.50MiB (rounded to 4718592)requested by op AddV2
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. 
Current allocation summary follows.
Current allocation summary follows.
2022-02-11 12:56:42.781251: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] BFCAllocator dump for GPU_0_bfc
2022-02-11 12:56:42.781263: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (256):  Total Chunks: 93, Chunks in use: 93. 23.2KiB allocated for chunks. 23.2KiB in use in bin. 11.3KiB client-requested in use in bin.
2022-02-11 12:56:42.781271: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (512):  Total Chunks: 40, Chunks in use: 40. 20.5KiB allocated for chunks. 20.5KiB in use in bin. 20.0KiB client-requested in use in bin.
2022-02-11 12:56:42.781279: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1024):     Total Chunks: 57, Chunks in use: 57. 58.2KiB allocated for chunks. 58.2KiB in use in bin. 57.0KiB client-requested in use in bin.
2022-02-11 12:56:42.781287: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2048):     Total Chunks: 65, Chunks in use: 65. 131.8KiB allocated for chunks. 131.8KiB in use in bin. 130.0KiB client-requested in use in bin.
2022-02-11 12:56:42.781296: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4096):     Total Chunks: 3, Chunks in use: 3. 18.2KiB allocated for chunks. 18.2KiB in use in bin. 14.8KiB client-requested in use in bin.
2022-02-11 12:56:42.781303: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8192):     Total Chunks: 2, Chunks in use: 2. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 13.5KiB client-requested in use in bin.
2022-02-11 12:56:42.781311: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16384):    Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin.
2022-02-11 12:56:42.781319: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (32768):    Total Chunks: 3, Chunks in use: 3. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 120.0KiB client-requested in use in bin.
2022-02-11 12:56:42.781348: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (65536):    Total Chunks: 1, Chunks in use: 1. 81.2KiB allocated for chunks. 81.2KiB in use in bin. 72.0KiB client-requested in use in bin.
2022-02-11 12:56:42.781363: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (131072):   Total Chunks: 5, Chunks in use: 3. 895.0KiB allocated for chunks. 540.0KiB in use in bin. 432.0KiB client-requested in use in bin.
2022-02-11 12:56:42.781376: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (262144):   Total Chunks: 7, Chunks in use: 5. 2.16MiB allocated for chunks. 1.41MiB in use in bin. 1.31MiB client-requested in use in bin.
2022-02-11 12:56:42.781390: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (524288):   Total Chunks: 6, Chunks in use: 4. 3.84MiB allocated for chunks. 2.44MiB in use in bin. 2.44MiB client-requested in use in bin.
2022-02-11 12:56:42.781413: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1048576):  Total Chunks: 6, Chunks in use: 4. 8.06MiB allocated for chunks. 5.81MiB in use in bin. 4.50MiB client-requested in use in bin.
2022-02-11 12:56:42.781431: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2097152):  Total Chunks: 8, Chunks in use: 6. 18.00MiB allocated for chunks. 13.50MiB in use in bin. 13.50MiB client-requested in use in bin.
2022-02-11 12:56:42.781445: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4194304):  Total Chunks: 7, Chunks in use: 7. 32.50MiB allocated for chunks. 32.50MiB in use in bin. 29.50MiB client-requested in use in bin.
2022-02-11 12:56:42.781460: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8388608):  Total Chunks: 10, Chunks in use: 10. 92.65MiB allocated for chunks. 92.65MiB in use in bin. 90.00MiB client-requested in use in bin.
2022-02-11 12:56:42.781474: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16777216):     Total Chunks: 1, Chunks in use: 1. 24.50MiB allocated for chunks. 24.50MiB in use in bin. 18.00MiB client-requested in use in bin.
2022-02-11 12:56:42.781487: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (33554432):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-02-11 12:56:42.781503: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (67108864):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-02-11 12:56:42.781516: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (134217728):    Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-02-11 12:56:42.781528: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (268435456):    Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-02-11 12:56:42.781540: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Bin for 4.50MiB was 4.00MiB, Chunk State: 
2022-02-11 12:56:42.781551: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 191987712
2022-02-11 12:56:42.781567: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0000 of size 1280 next 1
2022-02-11 12:56:42.781577: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0500 of size 256 next 2
2022-02-11 12:56:42.781587: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0600 of size 256 next 3
2022-02-11 12:56:42.781596: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0700 of size 256 next 4
2022-02-11 12:56:42.781606: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0800 of size 256 next 5
2022-02-11 12:56:42.781615: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0900 of size 256 next 8
2022-02-11 12:56:42.781624: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0a00 of size 256 next 9
2022-02-11 12:56:42.781633: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0b00 of size 256 next 10
2022-02-11 12:56:42.781641: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0c00 of size 256 next 11
2022-02-11 12:56:42.781651: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0d00 of size 256 next 12
2022-02-11 12:56:42.781660: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0e00 of size 256 next 13
2022-02-11 12:56:42.781669: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae0f00 of size 256 next 14
2022-02-11 12:56:42.781679: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1000 of size 256 next 15
2022-02-11 12:56:42.781703: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1100 of size 256 next 18
2022-02-11 12:56:42.781713: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1200 of size 256 next 19
2022-02-11 12:56:42.781723: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1300 of size 256 next 20
2022-02-11 12:56:42.781732: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1400 of size 256 next 21
2022-02-11 12:56:42.781742: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1500 of size 256 next 22
2022-02-11 12:56:42.781751: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1600 of size 256 next 23
2022-02-11 12:56:42.781763: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1700 of size 512 next 26
2022-02-11 12:56:42.781787: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1900 of size 512 next 27
2022-02-11 12:56:42.781795: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1b00 of size 512 next 28
2022-02-11 12:56:42.781805: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1d00 of size 512 next 29
2022-02-11 12:56:42.781814: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae1f00 of size 512 next 30
2022-02-11 12:56:42.781824: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2100 of size 256 next 31
2022-02-11 12:56:42.781833: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2200 of size 256 next 32
2022-02-11 12:56:42.781842: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2300 of size 512 next 33
2022-02-11 12:56:42.781851: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2500 of size 512 next 36
2022-02-11 12:56:42.781860: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2700 of size 512 next 37
2022-02-11 12:56:42.781869: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2900 of size 512 next 38
2022-02-11 12:56:42.781878: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2b00 of size 512 next 39
2022-02-11 12:56:42.781887: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2d00 of size 256 next 40
2022-02-11 12:56:42.781896: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2e00 of size 256 next 41
2022-02-11 12:56:42.781917: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae2f00 of size 1024 next 44
2022-02-11 12:56:42.781926: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae3300 of size 1024 next 45
2022-02-11 12:56:42.781937: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae3700 of size 1792 next 6
2022-02-11 12:56:42.781946: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae3e00 of size 1024 next 43
2022-02-11 12:56:42.781955: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae4200 of size 1024 next 7
2022-02-11 12:56:42.781964: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae4600 of size 1024 next 52
2022-02-11 12:56:42.781989: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae4a00 of size 2048 next 51
2022-02-11 12:56:42.781999: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae5200 of size 3840 next 47
2022-02-11 12:56:42.782008: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae6100 of size 256 next 48
2022-02-11 12:56:42.782017: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae6200 of size 256 next 49
2022-02-11 12:56:42.782026: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae6300 of size 1024 next 50
2022-02-11 12:56:42.782036: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae6700 of size 1024 next 53
2022-02-11 12:56:42.782045: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae6b00 of size 1024 next 54
2022-02-11 12:56:42.782054: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae6f00 of size 1024 next 55
2022-02-11 12:56:42.782063: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae7300 of size 1024 next 56
2022-02-11 12:56:42.782072: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae7700 of size 1024 next 57
2022-02-11 12:56:42.782081: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae7b00 of size 1024 next 59
2022-02-11 12:56:42.782091: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae7f00 of size 1024 next 60
2022-02-11 12:56:42.782100: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae8300 of size 1024 next 61
2022-02-11 12:56:42.782109: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae8700 of size 1024 next 62
2022-02-11 12:56:42.782118: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae8b00 of size 256 next 63
2022-02-11 12:56:42.782128: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae8c00 of size 256 next 64
2022-02-11 12:56:42.782137: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae8d00 of size 2048 next 67
2022-02-11 12:56:42.782146: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae9500 of size 2048 next 68
2022-02-11 12:56:42.782156: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ae9d00 of size 2048 next 69
2022-02-11 12:56:42.782165: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aea500 of size 2048 next 70
2022-02-11 12:56:42.782174: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aead00 of size 2048 next 71
2022-02-11 12:56:42.782183: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aeb500 of size 256 next 72
2022-02-11 12:56:42.782192: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aeb600 of size 256 next 73
2022-02-11 12:56:42.782202: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aeb700 of size 2048 next 74
2022-02-11 12:56:42.782211: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aebf00 of size 2048 next 77
2022-02-11 12:56:42.782220: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aec700 of size 2048 next 78
2022-02-11 12:56:42.782230: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aecf00 of size 2048 next 79
2022-02-11 12:56:42.782239: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aed700 of size 2048 next 80
2022-02-11 12:56:42.782249: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aedf00 of size 2048 next 81
2022-02-11 12:56:42.782260: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aee700 of size 2048 next 83
2022-02-11 12:56:42.782269: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aeef00 of size 2048 next 84
2022-02-11 12:56:42.782279: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aef700 of size 2048 next 85
2022-02-11 12:56:42.782288: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aeff00 of size 2048 next 86
2022-02-11 12:56:42.782297: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af0700 of size 2048 next 87
2022-02-11 12:56:42.782307: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af0f00 of size 2048 next 89
2022-02-11 12:56:42.782317: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af1700 of size 2048 next 90
2022-02-11 12:56:42.782327: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af1f00 of size 2048 next 91
2022-02-11 12:56:42.782336: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af2700 of size 2048 next 92
2022-02-11 12:56:42.782345: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af2f00 of size 2048 next 93
2022-02-11 12:56:42.782355: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af3700 of size 2048 next 95
2022-02-11 12:56:42.782364: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af3f00 of size 2048 next 96
2022-02-11 12:56:42.782373: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af4700 of size 2048 next 97
2022-02-11 12:56:42.782383: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af4f00 of size 2048 next 98
2022-02-11 12:56:42.782393: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af5700 of size 2048 next 99
2022-02-11 12:56:42.782412: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af5f00 of size 2048 next 101
2022-02-11 12:56:42.782423: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af6700 of size 2048 next 102
2022-02-11 12:56:42.782432: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af6f00 of size 2048 next 103
2022-02-11 12:56:42.782442: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af7700 of size 2048 next 104
2022-02-11 12:56:42.782451: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af7f00 of size 256 next 105
2022-02-11 12:56:42.782460: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af8000 of size 256 next 106
2022-02-11 12:56:42.782469: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af8100 of size 256 next 107
2022-02-11 12:56:42.782480: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af8200 of size 4096 next 110
2022-02-11 12:56:42.782490: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af9200 of size 256 next 111
2022-02-11 12:56:42.782500: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af9300 of size 256 next 112
2022-02-11 12:56:42.782624: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af9400 of size 2048 next 116
2022-02-11 12:56:42.782637: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703af9c00 of size 2048 next 117
2022-02-11 12:56:42.782647: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afa400 of size 2048 next 118
2022-02-11 12:56:42.782656: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afac00 of size 1024 next 120
2022-02-11 12:56:42.782666: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afb000 of size 1024 next 122
2022-02-11 12:56:42.782675: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afb400 of size 1024 next 123
2022-02-11 12:56:42.782684: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afb800 of size 1024 next 124
2022-02-11 12:56:42.782695: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afbc00 of size 1024 next 42
2022-02-11 12:56:42.782705: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afc000 of size 1024 next 216
2022-02-11 12:56:42.782715: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afc400 of size 1024 next 219
2022-02-11 12:56:42.782724: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afc800 of size 1024 next 220
2022-02-11 12:56:42.782734: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afcc00 of size 1024 next 221
2022-02-11 12:56:42.782743: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afd000 of size 1024 next 222
2022-02-11 12:56:42.782753: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afd400 of size 1024 next 224
2022-02-11 12:56:42.782762: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afd800 of size 1024 next 225
2022-02-11 12:56:42.782771: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afdc00 of size 1024 next 126
2022-02-11 12:56:42.782781: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afe000 of size 256 next 127
2022-02-11 12:56:42.782791: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afe100 of size 256 next 128
2022-02-11 12:56:42.782800: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afe200 of size 1024 next 130
2022-02-11 12:56:42.782810: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afe600 of size 1024 next 131
2022-02-11 12:56:42.782819: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afea00 of size 1024 next 132
2022-02-11 12:56:42.782829: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703afee00 of size 1024 next 133
2022-02-11 12:56:42.782838: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aff200 of size 512 next 134
2022-02-11 12:56:42.782848: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aff400 of size 512 next 136
2022-02-11 12:56:42.782857: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aff600 of size 512 next 137
2022-02-11 12:56:42.782866: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703aff800 of size 512 next 138
2022-02-11 12:56:42.782876: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703affa00 of size 512 next 139
2022-02-11 12:56:42.782886: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703affc00 of size 1024 next 140
2022-02-11 12:56:42.782896: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00000 of size 256 next 141
2022-02-11 12:56:42.782905: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00100 of size 256 next 142
2022-02-11 12:56:42.782915: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00200 of size 512 next 145
2022-02-11 12:56:42.782925: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00400 of size 512 next 146
2022-02-11 12:56:42.782934: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00600 of size 512 next 147
2022-02-11 12:56:42.782943: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00800 of size 512 next 148
2022-02-11 12:56:42.782953: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00a00 of size 256 next 149
2022-02-11 12:56:42.782977: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00b00 of size 256 next 150
2022-02-11 12:56:42.782986: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00c00 of size 256 next 152
2022-02-11 12:56:42.782995: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00d00 of size 256 next 153
2022-02-11 12:56:42.783005: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00e00 of size 256 next 114
2022-02-11 12:56:42.783016: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b00f00 of size 256 next 182
2022-02-11 12:56:42.783025: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01000 of size 256 next 155
2022-02-11 12:56:42.783034: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01100 of size 256 next 156
2022-02-11 12:56:42.783043: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01200 of size 256 next 157
2022-02-11 12:56:42.783052: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01300 of size 256 next 158
2022-02-11 12:56:42.783062: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01400 of size 256 next 159
2022-02-11 12:56:42.783071: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01500 of size 256 next 161
2022-02-11 12:56:42.783079: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01600 of size 256 next 162
2022-02-11 12:56:42.783089: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01700 of size 256 next 163
2022-02-11 12:56:42.783098: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01800 of size 256 next 164
2022-02-11 12:56:42.783107: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01900 of size 256 next 165
2022-02-11 12:56:42.783116: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01a00 of size 256 next 174
2022-02-11 12:56:42.783125: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01b00 of size 256 next 176
2022-02-11 12:56:42.783135: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01c00 of size 256 next 178
2022-02-11 12:56:42.783144: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01d00 of size 256 next 179
2022-02-11 12:56:42.783153: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b01e00 of size 1024 next 226
2022-02-11 12:56:42.783163: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b02200 of size 1024 next 227
2022-02-11 12:56:42.783172: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b02600 of size 1024 next 229
2022-02-11 12:56:42.783181: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b02a00 of size 1024 next 230
2022-02-11 12:56:42.783190: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b02e00 of size 1024 next 231
2022-02-11 12:56:42.783200: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b03200 of size 1024 next 232
2022-02-11 12:56:42.783209: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b03600 of size 1024 next 233
2022-02-11 12:56:42.783218: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b03a00 of size 2048 next 234
2022-02-11 12:56:42.783227: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b04200 of size 2048 next 237
2022-02-11 12:56:42.783236: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b04a00 of size 2048 next 238
2022-02-11 12:56:42.783245: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b05200 of size 2048 next 239
2022-02-11 12:56:42.783254: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b05a00 of size 2048 next 240
2022-02-11 12:56:42.783263: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b06200 of size 2048 next 243
2022-02-11 12:56:42.783272: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b06a00 of size 2048 next 244
2022-02-11 12:56:42.783281: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b07200 of size 2048 next 245
2022-02-11 12:56:42.783290: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b07a00 of size 2048 next 246
2022-02-11 12:56:42.783299: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b08200 of size 2048 next 247
2022-02-11 12:56:42.783325: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b08a00 of size 2048 next 249
2022-02-11 12:56:42.783335: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b09200 of size 2048 next 250
2022-02-11 12:56:42.783345: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b09a00 of size 2048 next 251
2022-02-11 12:56:42.783354: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0a200 of size 2048 next 252
2022-02-11 12:56:42.783364: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0aa00 of size 2048 next 253
2022-02-11 12:56:42.783373: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0b200 of size 2048 next 255
2022-02-11 12:56:42.783382: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0ba00 of size 2048 next 256
2022-02-11 12:56:42.783392: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0c200 of size 2048 next 257
2022-02-11 12:56:42.783411: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0ca00 of size 2048 next 258
2022-02-11 12:56:42.783423: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0d200 of size 2048 next 259
2022-02-11 12:56:42.783432: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0da00 of size 2048 next 261
2022-02-11 12:56:42.783441: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0e200 of size 2048 next 262
2022-02-11 12:56:42.783451: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0ea00 of size 2048 next 263
2022-02-11 12:56:42.783460: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0f200 of size 2048 next 264
2022-02-11 12:56:42.783469: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b0fa00 of size 2048 next 265
2022-02-11 12:56:42.783479: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b10200 of size 2048 next 266
2022-02-11 12:56:42.783488: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b10a00 of size 2048 next 267
2022-02-11 12:56:42.783498: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b11200 of size 2048 next 268
2022-02-11 12:56:42.783507: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b11a00 of size 2048 next 269
2022-02-11 12:56:42.783516: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b12200 of size 2048 next 270
2022-02-11 12:56:42.783526: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b12a00 of size 256 next 271
2022-02-11 12:56:42.783536: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b12b00 of size 256 next 272
2022-02-11 12:56:42.783545: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b1
---------------------------------------------------------------------------
ResourceExhaustedError                    Traceback (most recent call last)
/tmp/ipykernel_13584/3251730512.py in <module>
----> 1 detector = keras_ocr.detection.Detector(weights='clovaai_general')

/opt/conda/lib/python3.7/site-packages/keras_ocr/detection.py in __init__(self, weights, load_from_torch, optimizer, backbone_name)
    693             weights_path = None
    694         self.model = build_keras_model(
--> 695             weights_path=weights_path, backbone_name=backbone_name
    696         )
    697         self.model.compile(loss="mse", optimizer=optimizer)

/opt/conda/lib/python3.7/site-packages/keras_ocr/detection.py in build_keras_model(weights_path, backbone_name)
    355 
    356     if backbone_name == "vgg":
--> 357         s1, s2, s3, s4 = build_vgg_backbone(inputs)
    358     elif "efficientnet" in backbone_name.lower():
    359         s1, s2, s3, s4 = build_efficientnet_backbone(

/opt/conda/lib/python3.7/site-packages/keras_ocr/detection.py in build_vgg_backbone(inputs)
    318     x = make_vgg_block(x, filters=256, n=17, pooling=False, prefix="basenet.slice2")
    319     x = make_vgg_block(x, filters=256, n=20, pooling=True, prefix="basenet.slice3")
--> 320     x = make_vgg_block(x, filters=512, n=24, pooling=False, prefix="basenet.slice3")
    321     x = make_vgg_block(x, filters=512, n=27, pooling=False, prefix="basenet.slice3")
    322     x = make_vgg_block(x, filters=512, n=30, pooling=True, prefix="basenet.slice4")

/opt/conda/lib/python3.7/site-packages/keras_ocr/detection.py in make_vgg_block(x, filters, n, prefix, pooling)
     92         padding="same",
     93         name=f"{prefix}.{n}",
---> 94     )(x)
     95     x = keras.layers.BatchNormalization(
     96         momentum=0.1, epsilon=1e-5, axis=-1, name=f"{prefix}.{n+1}"

/opt/conda/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

/opt/conda/lib/python3.7/site-packages/keras/backend.py in random_uniform(self, shape, minval, maxval, dtype)
   1831     return tf.random.uniform(
   1832         shape=shape, minval=minval, maxval=maxval, dtype=dtype,
-> 1833         seed=self.make_legacy_seed())
   1834 
   1835   def truncated_normal(self, shape, mean=0., stddev=1., dtype=None):

ResourceExhaustedError: failed to allocate memory [Op:AddV2]
2c00 of size 256 next 274
2022-02-11 12:56:42.783555: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b12d00 of size 256 next 275
2022-02-11 12:56:42.783564: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b12e00 of size 256 next 276
2022-02-11 12:56:42.783573: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b12f00 of size 256 next 277
2022-02-11 12:56:42.783583: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13000 of size 256 next 279
2022-02-11 12:56:42.783592: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13100 of size 256 next 280
2022-02-11 12:56:42.783602: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13200 of size 256 next 281
2022-02-11 12:56:42.783611: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13300 of size 256 next 282
2022-02-11 12:56:42.783621: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13400 of size 512 next 283
2022-02-11 12:56:42.783630: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13600 of size 512 next 285
2022-02-11 12:56:42.783641: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13800 of size 512 next 286
2022-02-11 12:56:42.783651: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13a00 of size 512 next 287
2022-02-11 12:56:42.783660: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13c00 of size 512 next 288
2022-02-11 12:56:42.783669: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b13e00 of size 512 next 291
2022-02-11 12:56:42.783679: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b14000 of size 512 next 292
2022-02-11 12:56:42.783689: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b14200 of size 512 next 293
2022-02-11 12:56:42.783698: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b14400 of size 512 next 294
2022-02-11 12:56:42.783708: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b14600 of size 512 next 295
2022-02-11 12:56:42.783717: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b14800 of size 1024 next 296
2022-02-11 12:56:42.783727: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b14c00 of size 1024 next 298
2022-02-11 12:56:42.783738: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b15000 of size 1024 next 299
2022-02-11 12:56:42.783747: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b15400 of size 1024 next 300
2022-02-11 12:56:42.783756: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b15800 of size 1024 next 301
2022-02-11 12:56:42.783765: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b15c00 of size 1280 next 273
2022-02-11 12:56:42.783791: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b16100 of size 13056 next 160
2022-02-11 12:56:42.783800: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b19400 of size 83200 next 17
2022-02-11 12:56:42.783810: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b2d900 of size 49152 next 154
2022-02-11 12:56:42.783819: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b39900 of size 36864 next 167
2022-02-11 12:56:42.783829: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b42900 of size 61440 next 16
2022-02-11 12:56:42.783839: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b51900 of size 18432 next 175
2022-02-11 12:56:42.783848: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56100 of size 256 next 197
2022-02-11 12:56:42.783857: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56200 of size 256 next 198
2022-02-11 12:56:42.783867: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56300 of size 256 next 200
2022-02-11 12:56:42.783876: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56400 of size 256 next 201
2022-02-11 12:56:42.783886: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56500 of size 256 next 202
2022-02-11 12:56:42.784001: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56600 of size 256 next 203
2022-02-11 12:56:42.784014: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56700 of size 512 next 205
2022-02-11 12:56:42.784023: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56900 of size 512 next 206
2022-02-11 12:56:42.784032: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56b00 of size 512 next 207
2022-02-11 12:56:42.784041: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56d00 of size 512 next 208
2022-02-11 12:56:42.784050: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b56f00 of size 512 next 209
2022-02-11 12:56:42.784059: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b57100 of size 512 next 212
2022-02-11 12:56:42.784068: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b57300 of size 512 next 213
2022-02-11 12:56:42.784077: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b57500 of size 512 next 214
2022-02-11 12:56:42.784086: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b57700 of size 512 next 215
2022-02-11 12:56:42.784096: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b57900 of size 768 next 108
2022-02-11 12:56:42.784106: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b57c00 of size 11520 next 177
2022-02-11 12:56:42.784116: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b5a900 of size 258048 next 25
2022-02-11 12:56:42.784126: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703b99900 of size 294912 next 24
2022-02-11 12:56:42.784136: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be1900 of size 256 next 166
2022-02-11 12:56:42.784144: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be1a00 of size 256 next 168
2022-02-11 12:56:42.784153: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be1b00 of size 256 next 169
2022-02-11 12:56:42.784162: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be1c00 of size 256 next 170
2022-02-11 12:56:42.784172: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be1d00 of size 256 next 171
2022-02-11 12:56:42.784181: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be1e00 of size 256 next 172
2022-02-11 12:56:42.784190: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be1f00 of size 256 next 173
2022-02-11 12:56:42.784200: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2000 of size 256 next 180
2022-02-11 12:56:42.784209: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2100 of size 256 next 183
2022-02-11 12:56:42.784218: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2200 of size 256 next 184
2022-02-11 12:56:42.784227: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2300 of size 256 next 185
2022-02-11 12:56:42.784235: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2400 of size 256 next 186
2022-02-11 12:56:42.784244: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2500 of size 256 next 189
2022-02-11 12:56:42.784253: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2600 of size 256 next 190
2022-02-11 12:56:42.784263: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2700 of size 256 next 191
2022-02-11 12:56:42.784272: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2800 of size 256 next 193
2022-02-11 12:56:42.784281: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2900 of size 768 next 187
2022-02-11 12:56:42.784290: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2c00 of size 256 next 144
2022-02-11 12:56:42.784299: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2d00 of size 256 next 88
2022-02-11 12:56:42.784308: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2e00 of size 256 next 76
2022-02-11 12:56:42.784318: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be2f00 of size 256 next 188
2022-02-11 12:56:42.784328: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be3000 of size 6912 next 192
2022-02-11 12:56:42.784337: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be4b00 of size 7680 next 181
2022-02-11 12:56:42.784363: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703be6900 of size 147456 next 199
2022-02-11 12:56:42.784372: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 703c0a900 of size 225280 next 143
2022-02-11 12:56:42.784392: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703c41900 of size 294912 next 151
2022-02-11 12:56:42.784411: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 703c89900 of size 491520 next 35
2022-02-11 12:56:42.784421: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d01900 of size 294912 next 115
2022-02-11 12:56:42.784431: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d49900 of size 1024 next 302
2022-02-11 12:56:42.784441: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d49d00 of size 1024 next 305
2022-02-11 12:56:42.784450: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d4a100 of size 1024 next 306
2022-02-11 12:56:42.784459: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d4a500 of size 1024 next 307
2022-02-11 12:56:42.784469: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d4a900 of size 1024 next 308
2022-02-11 12:56:42.784479: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d4ad00 of size 1024 next 310
2022-02-11 12:56:42.784488: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d4b100 of size 1024 next 311
2022-02-11 12:56:42.784498: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d4b500 of size 1024 next 312
2022-02-11 12:56:42.784507: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d4b900 of size 1024 next 313
2022-02-11 12:56:42.784516: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 703d4bd00 of size 138240 next 278
2022-02-11 12:56:42.784526: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d6d900 of size 147456 next 34
2022-02-11 12:56:42.784535: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703d91900 of size 589824 next 194
2022-02-11 12:56:42.784545: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703e21900 of size 786432 next 125
2022-02-11 12:56:42.784555: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 703ee1900 of size 1769472 next 129
2022-02-11 12:56:42.784565: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 704091900 of size 294912 next 204
2022-02-11 12:56:42.784575: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7040d9900 of size 884736 next 135
2022-02-11 12:56:42.784584: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7041b1900 of size 1179648 next 195
2022-02-11 12:56:42.784595: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7042d1900 of size 6291456 next 58
2022-02-11 12:56:42.784604: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7048d1900 of size 2359296 next 46
2022-02-11 12:56:42.784614: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 704b11900 of size 2359296 next 66
2022-02-11 12:56:42.784624: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 704d51900 of size 4718592 next 65
2022-02-11 12:56:42.784634: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7051d1900 of size 4194304 next 113
2022-02-11 12:56:42.784643: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7055d1900 of size 294912 next 284
2022-02-11 12:56:42.784653: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 705619900 of size 294912 next 211
2022-02-11 12:56:42.784663: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 705661900 of size 589824 next 210
2022-02-11 12:56:42.784687: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7056f1900 of size 589824 next 290
2022-02-11 12:56:42.784696: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 705781900 of size 589824 next 289
2022-02-11 12:56:42.784706: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 705811900 of size 1179648 next 218
2022-02-11 12:56:42.784715: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 705931900 of size 1179648 next 217
2022-02-11 12:56:42.784724: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 705a51900 of size 1179648 next 297
2022-02-11 12:56:42.784734: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 705b71900 of size 1966080 next 121
2022-02-11 12:56:42.784743: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 705d51900 of size 9437184 next 75
2022-02-11 12:56:42.784752: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 706651900 of size 9437184 next 82
2022-02-11 12:56:42.784762: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 706f51900 of size 25690112 next 94
2022-02-11 12:56:42.784772: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7087d1900 of size 9437184 next 100
2022-02-11 12:56:42.784780: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7090d1900 of size 9437184 next 196
2022-02-11 12:56:42.784789: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7099d1900 of size 9437184 next 109
2022-02-11 12:56:42.784799: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70a2d1900 of size 4718592 next 119
2022-02-11 12:56:42.784808: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70a751900 of size 2359296 next 223
2022-02-11 12:56:42.784817: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70a991900 of size 2359296 next 228
2022-02-11 12:56:42.784826: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 70abd1900 of size 2359296 next 309
2022-02-11 12:56:42.784834: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70ae11900 of size 2359296 next 304
2022-02-11 12:56:42.784844: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70b051900 of size 2359296 next 303
2022-02-11 12:56:42.784853: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 70b291900 of size 2359296 next 236
2022-02-11 12:56:42.784862: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70b4d1900 of size 4718592 next 235
2022-02-11 12:56:42.784871: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70b951900 of size 4718592 next 314
2022-02-11 12:56:42.784881: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70bdd1900 of size 4718592 next 242
2022-02-11 12:56:42.784890: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70c251900 of size 9437184 next 241
2022-02-11 12:56:42.784899: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70cb51900 of size 9437184 next 248
2022-02-11 12:56:42.784909: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70d451900 of size 9437184 next 254
2022-02-11 12:56:42.784919: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70dd51900 of size 9437184 next 260
2022-02-11 12:56:42.784929: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 70e651900 of size 12216064 next 18446744073709551615
2022-02-11 12:56:42.784938: I tensorflow/core/common_runtime/bfc_allocator.cc:1071]      Summary of in-use Chunks by size: 
2022-02-11 12:56:42.784952: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 93 Chunks of size 256 totalling 23.2KiB
2022-02-11 12:56:42.784965: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 38 Chunks of size 512 totalling 19.0KiB
2022-02-11 12:56:42.784976: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 768 totalling 1.5KiB
2022-02-11 12:56:42.784987: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 54 Chunks of size 1024 totalling 54.0KiB
2022-02-11 12:56:42.784997: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 1280 totalling 2.5KiB
2022-02-11 12:56:42.785008: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1792 totalling 1.8KiB
2022-02-11 12:56:42.785019: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 64 Chunks of size 2048 totalling 128.0KiB
2022-02-11 12:56:42.785029: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 3840 totalling 3.8KiB
2022-02-11 12:56:42.785040: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 4096 totalling 4.0KiB
2022-02-11 12:56:42.785051: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 6912 totalling 6.8KiB
2022-02-11 12:56:42.785077: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 7680 totalling 7.5KiB
2022-02-11 12:56:42.785088: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 11520 totalling 11.2KiB
2022-02-11 12:56:42.785098: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 13056 totalling 12.8KiB
2022-02-11 12:56:42.785108: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 18432 totalling 18.0KiB
2022-02-11 12:56:42.785119: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 36864 totalling 36.0KiB
2022-02-11 12:56:42.785129: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 49152 totalling 48.0KiB
2022-02-11 12:56:42.785139: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 61440 totalling 60.0KiB
2022-02-11 12:56:42.785151: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 83200 totalling 81.2KiB
2022-02-11 12:56:42.785162: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 147456 totalling 288.0KiB
2022-02-11 12:56:42.785172: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 258048 totalling 252.0KiB
2022-02-11 12:56:42.785183: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 5 Chunks of size 294912 totalling 1.41MiB
2022-02-11 12:56:42.785193: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 589824 totalling 1.69MiB
2022-02-11 12:56:42.785204: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 786432 totalling 768.0KiB
2022-02-11 12:56:42.785215: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 1179648 totalling 2.25MiB
2022-02-11 12:56:42.785224: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1769472 totalling 1.69MiB
2022-02-11 12:56:42.785234: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1966080 totalling 1.88MiB
2022-02-11 12:56:42.785245: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 6 Chunks of size 2359296 totalling 13.50MiB
2022-02-11 12:56:42.785255: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 4194304 totalling 4.00MiB
2022-02-11 12:56:42.785266: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 5 Chunks of size 4718592 totalling 22.50MiB
2022-02-11 12:56:42.785276: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 6291456 totalling 6.00MiB
2022-02-11 12:56:42.785391: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 9 Chunks of size 9437184 totalling 81.00MiB
2022-02-11 12:56:42.785429: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 12216064 totalling 11.65MiB
2022-02-11 12:56:42.785457: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 25690112 totalling 24.50MiB
2022-02-11 12:56:42.785469: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 173.84MiB
2022-02-11 12:56:42.785480: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 191987712 memory_limit_: 213319680 available bytes: 21331968 curr_region_allocation_bytes_: 426639360
2022-02-11 12:56:42.785497: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats: 
Limit:                       213319680
InUse:                       182285312
MaxInUse:                    183959040
NumAllocs:                         584
MaxAllocSize:                 25690112
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2022-02-11 12:56:42.785525: W tensorflow/core/common_runtime/bfc_allocator.cc:474] ***************************************xxx*********************************************************x
2022-02-11 12:56:42.785564: W tensorflow/core/framework/op_kernel.cc:1733] RESOURCE_EXHAUSTED: failed to allocate memory
kovalepskaia commented 1 year ago

os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'