Resource Exhausted even on using small batch size, when training own dataset.

I am using the yolo.cfg to train my own dataset of 305 images. I used epoch of 200 and batch size 5, but I am still getting Resource exhausted error on training. I even tried using less gpu like 0.3/0.2, but same result. It works fine when training on cpu though. I have NVidia GForce 940 mx with memory of 2gb. Is this due to less gpu memory? here is the error I got: ``2018-09-18 03:28:10.143931: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2018-09-18 03:28:10.593199: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1405] Found device 0 with properties: name: GeForce 940MX major: 5 minor: 0 memoryClockRate(GHz): 1.2415 pciBusID: 0000:01:00.0 totalMemory: 2.00GiB freeMemory: 1.65GiB 2018-09-18 03:28:10.601315: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0 2018-09-18 03:28:11.794453: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-09-18 03:28:11.799661: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0 2018-09-18 03:28:11.802070: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N 2018-09-18 03:28:11.805159: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 409 MB memory) -> physical GPU (device: 0, name: GeForce 940MX, pci bus id: 0000:01:00.0, compute capability: 5.0) 2018-09-18 03:28:36.219172: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.0KiB. Current allocation summary follows. 2018-09-18 03:28:36.229659: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (256): Total Chunks: 24, Chunks in use: 24. 6.0KiB allocated for chunks. 6.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2018-09-18 03:28:36.236337: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (512): Total Chunks: 16, Chunks in use: 16. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin. 2018-09-18 03:28:36.243471: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1024): Total Chunks: 33, Chunks in use: 33. 33.3KiB allocated for chunks. 33.3KiB in use in bin. 33.0KiB client-requested in use in bin. 2018-09-18 03:28:36.250885: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2048): Total Chunks: 43, Chunks in use: 43. 90.5KiB allocated for chunks. 90.5KiB in use in bin. 90.1KiB client-requested in use in bin. 2018-09-18 03:28:36.257408: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4096): Total Chunks: 24, Chunks in use: 24. 96.0KiB allocated for chunks. 96.0KiB in use in bin. 96.0KiB client-requested in use in bin. 2018-09-18 03:28:36.263236: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-18 03:28:36.269794: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-18 03:28:36.275503: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-18 03:28:36.281118: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (65536): Total Chunks: 3, Chunks in use: 3. 216.0KiB allocated for chunks. 216.0KiB in use in bin. 216.0KiB client-requested in use in bin. 2018-09-18 03:28:36.287652: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (131072): Total Chunks: 6, Chunks in use: 6. 768.0KiB allocated for chunks. 768.0KiB in use in bin. 768.0KiB client-requested in use in bin. 2018-09-18 03:28:36.293821: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (262144): Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin. 2018-09-18 03:28:36.300301: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (524288): Total Chunks: 4, Chunks in use: 4. 2.00MiB allocated for chunks. 2.00MiB in use in bin. 2.00MiB client-requested in use in bin. 2018-09-18 03:28:36.306643: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1048576): Total Chunks: 4, Chunks in use: 4. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin. 2018-09-18 03:28:36.313059: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2097152): Total Chunks: 4, Chunks in use: 4. 8.00MiB allocated for chunks. 8.00MiB in use in bin. 8.00MiB client-requested in use in bin. 2018-09-18 03:28:36.318684: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4194304): Total Chunks: 5, Chunks in use: 5. 22.50MiB allocated for chunks. 22.50MiB in use in bin. 22.50MiB client-requested in use in bin. 2018-09-18 03:28:36.325114: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-18 03:28:36.331542: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16777216): Total Chunks: 5, Chunks in use: 5. 90.00MiB allocated for chunks. 90.00MiB in use in bin. 90.00MiB client-requested in use in bin. 2018-09-18 03:28:36.338247: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (33554432): Total Chunks: 7, Chunks in use: 7. 281.13MiB allocated for chunks. 281.13MiB in use in bin. 279.00MiB client-requested in use in bin. 2018-09-18 03:28:36.343919: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-18 03:28:36.350749: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-18 03:28:36.356587: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-18 03:28:36.362120: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:646] Bin for 4.0KiB was 4.0KiB, Chunk State: 2018-09-18 03:28:36.366811: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501180000 of size 1280 2018-09-18 03:28:36.370118: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501180500 of size 256 2018-09-18 03:28:36.373277: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501180600 of size 256 2018-09-18 03:28:36.377499: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501180700 of size 256 2018-09-18 03:28:36.381288: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501180800 of size 256 2018-09-18 03:28:36.385021: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501180900 of size 256 2018-09-18 03:28:36.388176: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501180A00 of size 256 2018-09-18 03:28:36.391426: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501180B00 of size 3584 2018-09-18 03:28:36.395471: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501181900 of size 3584 2018-09-18 03:28:36.399401: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501182700 of size 3584 2018-09-18 03:28:36.403239: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501183500 of size 256 2018-09-18 03:28:36.406390: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501183600 of size 256 2018-09-18 03:28:36.410352: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501183700 of size 512 2018-09-18 03:28:36.413807: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501183900 of size 512 2018-09-18 03:28:36.417316: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501183B00 of size 512 2018-09-18 03:28:36.421236: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501183D00 of size 512 2018-09-18 03:28:36.424509: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501183F00 of size 512 2018-09-18 03:28:36.428411: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501184100 of size 512 2018-09-18 03:28:36.432400: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501184300 of size 294912 2018-09-18 03:28:36.435639: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CC300 of size 512 2018-09-18 03:28:36.439623: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CC500 of size 512 2018-09-18 03:28:36.443043: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CC700 of size 1024 2018-09-18 03:28:36.450507: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CCB00 of size 1024 2018-09-18 03:28:36.455050: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CCF00 of size 1024 2018-09-18 03:28:36.459135: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CD300 of size 1024 2018-09-18 03:28:36.462476: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CD700 of size 1024 2018-09-18 03:28:36.465895: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CDB00 of size 1024 2018-09-18 03:28:36.469919: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005011CDF00 of size 1179648 2018-09-18 03:28:36.473169: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EDF00 of size 1024 2018-09-18 03:28:36.476923: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EE300 of size 1024 2018-09-18 03:28:36.480688: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EE700 of size 512 2018-09-18 03:28:36.484236: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EE900 of size 512 2018-09-18 03:28:36.488169: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EEB00 of size 512 2018-09-18 03:28:36.491536: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EED00 of size 512 2018-09-18 03:28:36.495419: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EEF00 of size 512 2018-09-18 03:28:36.499184: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EF100 of size 512 2018-09-18 03:28:36.503129: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005012EF300 of size 131072 2018-09-18 03:28:36.506354: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050130F300 of size 131072 2018-09-18 03:28:36.509703: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050132F300 of size 131072 2018-09-18 03:28:36.514108: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050134F300 of size 512 2018-09-18 03:28:36.517619: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050134F500 of size 512 2018-09-18 03:28:36.521614: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050134F700 of size 1024 2018-09-18 03:28:36.524993: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050134FB00 of size 1024 2018-09-18 03:28:36.528172: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050134FF00 of size 1024 2018-09-18 03:28:36.532724: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501350300 of size 1024 2018-09-18 03:28:36.535892: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501350700 of size 1024 2018-09-18 03:28:36.539747: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501350B00 of size 1024 2018-09-18 03:28:36.542934: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501350F00 of size 1179648 2018-09-18 03:28:36.546410: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501470F00 of size 1179648 2018-09-18 03:28:36.551806: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501590F00 of size 1179648 2018-09-18 03:28:36.555300: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B0F00 of size 1024 2018-09-18 03:28:36.559205: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B1300 of size 1024 2018-09-18 03:28:36.562546: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B1700 of size 2048 2018-09-18 03:28:36.566935: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B1F00 of size 2048 2018-09-18 03:28:36.570171: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B2700 of size 2048 2018-09-18 03:28:36.573333: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B2F00 of size 2048 2018-09-18 03:28:36.577157: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B3700 of size 2048 2018-09-18 03:28:36.580876: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B3F00 of size 2048 2018-09-18 03:28:36.585068: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005016B4700 of size 4718592 2018-09-18 03:28:36.588330: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B34700 of size 2048 2018-09-18 03:28:36.591607: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B34F00 of size 2048 2018-09-18 03:28:36.596870: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B35700 of size 1024 2018-09-18 03:28:36.601593: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B35B00 of size 1024 2018-09-18 03:28:36.605996: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B35F00 of size 1024 2018-09-18 03:28:36.609241: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B36300 of size 1024 2018-09-18 03:28:36.613255: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B36700 of size 1024 2018-09-18 03:28:36.616749: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B36B00 of size 1024 2018-09-18 03:28:36.620806: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501B36F00 of size 524288 2018-09-18 03:28:36.624013: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BB6F00 of size 1024 2018-09-18 03:28:36.627257: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BB7300 of size 1024 2018-09-18 03:28:36.632493: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BB7700 of size 2048 2018-09-18 03:28:36.635931: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BB7F00 of size 2048 2018-09-18 03:28:36.640002: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BB8700 of size 2048 2018-09-18 03:28:36.643189: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BB8F00 of size 2048 2018-09-18 03:28:36.646662: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BB9700 of size 2048 2018-09-18 03:28:36.651457: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BB9F00 of size 2048 2018-09-18 03:28:36.654637: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000501BBA700 of size 4718592 2018-09-18 03:28:36.657866: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203A700 of size 2048 2018-09-18 03:28:36.662078: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203AF00 of size 2048 2018-09-18 03:28:36.665790: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203B700 of size 1024 2018-09-18 03:28:36.670046: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203BB00 of size 1024 2018-09-18 03:28:36.673224: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203BF00 of size 1024 2018-09-18 03:28:36.677127: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203C300 of size 1024 2018-09-18 03:28:36.680910: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203C700 of size 1024 2018-09-18 03:28:36.684177: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203CB00 of size 1024 2018-09-18 03:28:36.688141: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050203CF00 of size 524288 2018-09-18 03:28:36.691477: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005020BCF00 of size 524288 2018-09-18 03:28:36.695494: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050213CF00 of size 524288 2018-09-18 03:28:36.699421: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021BCF00 of size 1024 2018-09-18 03:28:36.702682: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021BD300 of size 1024 2018-09-18 03:28:36.706589: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021BD700 of size 2048 2018-09-18 03:28:36.710162: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021BDF00 of size 2048 2018-09-18 03:28:36.714711: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021BE700 of size 2048 2018-09-18 03:28:36.717895: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021BEF00 of size 2048 2018-09-18 03:28:36.721848: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021BF700 of size 2048 2018-09-18 03:28:36.725019: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021BFF00 of size 2048 2018-09-18 03:28:36.728342: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005021C0700 of size 4718592 2018-09-18 03:28:36.733419: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502640700 of size 4718592 2018-09-18 03:28:36.736676: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502AC0700 of size 4718592 2018-09-18 03:28:36.740780: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F40700 of size 2048 2018-09-18 03:28:36.744245: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F40F00 of size 2048 2018-09-18 03:28:36.747872: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F41700 of size 256 2018-09-18 03:28:36.751824: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F41800 of size 256 2018-09-18 03:28:36.755029: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F41900 of size 256 2018-09-18 03:28:36.758869: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F41A00 of size 256 2018-09-18 03:28:36.762267: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F41B00 of size 256 2018-09-18 03:28:36.766053: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F41C00 of size 256 2018-09-18 03:28:36.770075: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F41D00 of size 73728 2018-09-18 03:28:36.773287: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F53D00 of size 73728 2018-09-18 03:28:36.777072: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F65D00 of size 73728 2018-09-18 03:28:36.780839: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F77D00 of size 256 2018-09-18 03:28:36.784371: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F77E00 of size 256 2018-09-18 03:28:36.788302: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F77F00 of size 4096 2018-09-18 03:28:36.792182: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F78F00 of size 4096 2018-09-18 03:28:36.796156: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000502F79F00 of size 18874368 2018-09-18 03:28:36.800221: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000504179F00 of size 4096 2018-09-18 03:28:36.804012: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050417AF00 of size 4096 2018-09-18 03:28:36.807166: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050417BF00 of size 2048 2018-09-18 03:28:36.810420: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050417C700 of size 2048 2018-09-18 03:28:36.819715: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050417CF00 of size 2048 2018-09-18 03:28:36.823105: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050417D700 of size 2048 2018-09-18 03:28:36.826960: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050417DF00 of size 2048 2018-09-18 03:28:36.831219: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050417E700 of size 2048 2018-09-18 03:28:36.835132: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050417EF00 of size 2097152 2018-09-18 03:28:36.838411: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050437EF00 of size 2048 2018-09-18 03:28:36.841700: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050437F700 of size 2048 2018-09-18 03:28:36.845733: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050437FF00 of size 4096 2018-09-18 03:28:36.849582: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000504380F00 of size 4096 2018-09-18 03:28:36.853627: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000504381F00 of size 18874368 2018-09-18 03:28:36.856863: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505581F00 of size 4096 2018-09-18 03:28:36.860066: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505582F00 of size 4096 2018-09-18 03:28:36.864339: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505583F00 of size 2048 2018-09-18 03:28:36.867850: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505584700 of size 2048 2018-09-18 03:28:36.871757: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505584F00 of size 2048 2018-09-18 03:28:36.874923: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505585700 of size 2048 2018-09-18 03:28:36.878082: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505585F00 of size 2048 2018-09-18 03:28:36.883230: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505586700 of size 2048 2018-09-18 03:28:36.886433: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505586F00 of size 2097152 2018-09-18 03:28:36.890527: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505786F00 of size 2097152 2018-09-18 03:28:36.894044: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505986F00 of size 2097152 2018-09-18 03:28:36.898430: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505B86F00 of size 2048 2018-09-18 03:28:36.901898: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505B87700 of size 2048 2018-09-18 03:28:36.908207: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505B87F00 of size 4096 2018-09-18 03:28:36.913995: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505B88F00 of size 4096 2018-09-18 03:28:36.919933: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000505B89F00 of size 18874368 2018-09-18 03:28:36.923200: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000506D89F00 of size 18874368 2018-09-18 03:28:36.927034: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000507F89F00 of size 18874368 2018-09-18 03:28:36.931129: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000509189F00 of size 4096 2018-09-18 03:28:36.935465: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050918AF00 of size 4096 2018-09-18 03:28:36.938668: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050918BF00 of size 4096 2018-09-18 03:28:36.942132: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050918CF00 of size 4096 2018-09-18 03:28:36.946119: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050918DF00 of size 37748736 2018-09-18 03:28:36.950012: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050B58DF00 of size 4096 2018-09-18 03:28:36.953987: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050B58EF00 of size 4096 2018-09-18 03:28:36.957162: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050B58FF00 of size 4096 2018-09-18 03:28:36.960483: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050B590F00 of size 4096 2018-09-18 03:28:36.969934: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050B591F00 of size 37748736 2018-09-18 03:28:36.973219: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050D991F00 of size 37748736 2018-09-18 03:28:36.977062: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000050FD91F00 of size 37748736 2018-09-18 03:28:36.981026: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512191F00 of size 4096 2018-09-18 03:28:36.984895: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512192F00 of size 4096 2018-09-18 03:28:36.988068: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512193F00 of size 256 2018-09-18 03:28:36.991304: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512194000 of size 256 2018-09-18 03:28:36.995364: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512194100 of size 256 2018-09-18 03:28:36.999281: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512194200 of size 256 2018-09-18 03:28:37.003142: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512194300 of size 256 2018-09-18 03:28:37.006291: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512194400 of size 256 2018-09-18 03:28:37.009495: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000512194500 of size 131072 2018-09-18 03:28:37.013713: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121B4500 of size 131072 2018-09-18 03:28:37.017337: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121D4500 of size 131072 2018-09-18 03:28:37.021391: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121F4500 of size 256 2018-09-18 03:28:37.024518: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121F4600 of size 256 2018-09-18 03:28:37.027657: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121F4700 of size 4096 2018-09-18 03:28:37.033756: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121F5700 of size 4096 2018-09-18 03:28:37.036930: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121F6700 of size 4096 2018-09-18 03:28:37.041038: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121F7700 of size 4096 2018-09-18 03:28:37.044198: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000005121F8700 of size 47185920 2018-09-18 03:28:37.048470: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000514EF8700 of size 47185920 2018-09-18 03:28:37.051793: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000517BF8700 of size 49418752 2018-09-18 03:28:37.055432: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:671] Summary of in-use Chunks by size: 2018-09-18 03:28:37.059451: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 24 Chunks of size 256 totalling 6.0KiB 2018-09-18 03:28:37.062779: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 16 Chunks of size 512 totalling 8.0KiB 2018-09-18 03:28:37.067377: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 32 Chunks of size 1024 totalling 32.0KiB 2018-09-18 03:28:37.070736: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB 2018-09-18 03:28:37.073906: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 40 Chunks of size 2048 totalling 80.0KiB 2018-09-18 03:28:37.077845: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 3 Chunks of size 3584 totalling 10.5KiB 2018-09-18 03:28:37.081676: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 24 Chunks of size 4096 totalling 96.0KiB 2018-09-18 03:28:37.085943: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 3 Chunks of size 73728 totalling 216.0KiB 2018-09-18 03:28:37.089253: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 6 Chunks of size 131072 totalling 768.0KiB 2018-09-18 03:28:37.092529: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 294912 totalling 288.0KiB 2018-09-18 03:28:37.096874: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 524288 totalling 2.00MiB 2018-09-18 03:28:37.100829: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 1179648 totalling 4.50MiB 2018-09-18 03:28:37.104749: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 2097152 totalling 8.00MiB 2018-09-18 03:28:37.108080: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 5 Chunks of size 4718592 totalling 22.50MiB 2018-09-18 03:28:37.112297: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 5 Chunks of size 18874368 totalling 90.00MiB 2018-09-18 03:28:37.116067: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 37748736 totalling 144.00MiB 2018-09-18 03:28:37.119507: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 47185920 totalling 90.00MiB 2018-09-18 03:28:37.123602: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 49418752 totalling 47.13MiB 2018-09-18 03:28:37.126914: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:678] Sum Total of in-use chunks: 409.60MiB 2018-09-18 03:28:37.131623: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats: Limit: 429496729 InUse: 429496576 MaxInUse: 429496576 NumAllocs: 179 MaxAllocSize: 49418752

2018-09-18 03:28:37.138062: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] **** 2018-09-18 03:28:37.143645: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1252] OP_REQUIRES failed at constant_op.cc:75 : Resource exhausted: OOM when allocating tensor of shape [1024] and type float 2018-09-18 03:28:37.196100: E T:\src\github\tensorflow\tensorflow\core\common_runtime\executor.cc:697] Executor failed to create kernel. Resource exhausted: OOM when allocating tensor of shape [1024] and type float [[Node: 50-convolutional/moving_mean/Initializer/Const = Const[_class=["loc:@50-convolutional/moving_mean/Assign"], dtype=DT_FLOAT, value=Tensor<type: float shape: [1024] values: -0.196210042 1.79337442 -0.631373823...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]] Traceback (most recent call last): File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1278, in _do_call return fn(*args) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1263, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [1024] and type float [[Node: 50-convolutional/moving_mean/Initializer/Const = Const[_class=["loc:@50-convolutional/moving_mean/Assign"], dtype=DT_FLOAT, value=Tensor<type: float shape: [1024] values: -0.196210042 1.79337442 -0.631373823...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "flow", line 6, in cliHandler(sys.argv) File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\cli.py", line 26, in cliHandler tfnet = TFNet(FLAGS) File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\net\build.py", line 76, in init self.setup_meta_ops() File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\net\build.py", line 146, in setup_meta_ops self.sess.run(tf.global_variables_initializer()) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 877, in run run_metadata_ptr) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1100, in _run feed_dict_tensor, options, run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1272, in _do_run run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1291, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [1024] and type float [[Node: 50-convolutional/moving_mean/Initializer/Const = Const[_class=["loc:@50-convolutional/moving_mean/Assign"], dtype=DT_FLOAT, value=Tensor<type: float shape: [1024] values: -0.196210042 1.79337442 -0.631373823...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op '50-convolutional/moving_mean/Initializer/Const', defined at: File "flow", line 6, in cliHandler(sys.argv) File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\cli.py", line 26, in cliHandler tfnet = TFNet(FLAGS) File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\net\build.py", line 75, in init self.build_forward() File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\net\build.py", line 115, in build_forward state = op_create(args) File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\net\ops__init__.py", line 27, in op_create return op_types[layer_type](args) File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\net\ops\baseop.py", line 42, in init self.forward() File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\net\ops\convolution.py", line 73, in forward temp = self.batchnorm(self.lay, temp) File "A:\Code\Machine Learning\Vehicle Detection\darkflow-master\darkflow\net\ops\convolution.py", line 90, in batchnorm return slim.batch_norm(inp, args) File "C:\Python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(*args, *current_args) File "C:\Python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 596, in batch_norm scope=scope) File "C:\Python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 353, in _fused_batch_norm collections=moving_mean_collections) File "C:\Python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(args, current_args) File "C:\Python36\lib\site-packages\tensorflow\contrib\framework\python\ops\variables.py", line 297, in model_variable use_resource=use_resource) File "C:\Python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(*args, current_args) File "C:\Python36\lib\site-packages\tensorflow\contrib\framework\python\ops\variables.py", line 252, in variable use_resource=use_resource) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1467, in get_variable aggregation=aggregation) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1217, in get_variable aggregation=aggregation) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 527, in get_variable aggregation=aggregation) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 481, in _true_getter aggregation=aggregation) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 903, in _get_single_variable aggregation=aggregation) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2443, in variable aggregation=aggregation) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2425, in previous_getter = lambda kwargs: default_variable_creator(None, *kwargs) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2406, in default_variable_creator constraint=constraint) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variables.py", line 259, in init constraint=constraint) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variables.py", line 368, in _init_from_args initial_value(), name="initial_value", dtype=dtype) File "C:\Python36\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 885, in shape.as_list(), dtype=dtype, partition_info=partition_info) File "C:\Python36\lib\site-packages\tensorflow\python\ops\init_ops.py", line 219, in call self.value, dtype=dtype, shape=shape, verify_shape=verify_shape) File "C:\Python36\lib\site-packages\tensorflow\python\framework\constant_op.py", line 202, in constant name=name).outputs[0] File "C:\Python36\lib\site-packages\tensorflow\python\util\deprecation.py", line 454, in new_func return func(args, **kwargs) File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3155, in create_op op_def=op_def) File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor of shape [1024] and type float [[Node: 50-convolutional/moving_mean/Initializer/Const = Const[_class=["loc:@50-convolutional/moving_mean/Assign"], dtype=DT_FLOAT, value=Tensor<type: float shape: [1024] values: -0.196210042 1.79337442 -0.631373823...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

thtrieu / darkflow

Resource Exhausted even on using small batch size, when training own dataset. #899

Training