Hi,
when I tried to run the object tracker with
python object_tracker.py --video 0 --output ./data/video/results.avi
unfortunatelly an error occurs. You can find the whole output below.
Here are some of the errors which are listed within the complete output below:
2020-09-15 16:27:45.302542: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
...
2020-09-15 16:28:22.690033: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
...
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_49/kernel/Initializer/random_uniform/
I'm using a Jetson Nano with JetPack 4.3 and Tensorflow version 2.1.0.
I would appreciate some good advice.
BR
Complete output text
2020-09-15 16:27:28.567398: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:33.539154: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-09-15 16:27:33.571472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-09-15 16:27:44.860933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-15 16:27:44.902688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:44.902874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-15 16:27:44.903030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:44.903205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-15 16:27:44.984059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-15 16:27:45.093582: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-15 16:27:45.206979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-15 16:27:45.270792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-15 16:27:45.271229: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-15 16:27:45.271733: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.272165: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.272330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-15 16:27:45.302542: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-09-15 16:27:45.303124: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11c62ec0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-15 16:27:45.303181: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-09-15 16:27:45.405645: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.405961: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11cb26c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-15 16:27:45.406032: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-09-15 16:27:45.406766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.406916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-15 16:27:45.407163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:45.407278: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-15 16:27:45.407412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-15 16:27:45.407530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-15 16:27:45.407637: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-15 16:27:45.407744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-15 16:27:45.407832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-15 16:27:45.408170: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.408587: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.408671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-15 16:27:45.408832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:55.785902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-15 16:27:55.786029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-09-15 16:27:55.786074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-09-15 16:27:55.814766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:55.815485: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:55.815804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 183 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-09-15 16:27:57.130111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.146867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-15 16:27:57.162859: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:57.194069: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-15 16:27:57.311603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-15 16:27:57.312279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-15 16:27:57.322676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-15 16:27:57.323023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-15 16:27:57.357181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-15 16:27:57.357784: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.358266: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.358402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-15 16:27:57.358507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-15 16:27:57.358562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-09-15 16:27:57.358607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-09-15 16:27:57.359069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.359611: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.359806: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 183 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-09-15 16:28:22.589148: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 18.00MiB (rounded to 18874368). Current allocation summary follows.
2020-09-15 16:28:22.603518: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
2020-09-15 16:28:22.603676: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): Total Chunks: 25, Chunks in use: 25. 6.2KiB allocated for chunks. 6.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2020-09-15 16:28:22.603727: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): Total Chunks: 44, Chunks in use: 44. 22.0KiB allocated for chunks. 22.0KiB in use in bin. 22.0KiB client-requested in use in bin.
2020-09-15 16:28:22.603769: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): Total Chunks: 69, Chunks in use: 69. 69.2KiB allocated for chunks. 69.2KiB in use in bin. 69.0KiB client-requested in use in bin.
2020-09-15 16:28:22.603811: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): Total Chunks: 49, Chunks in use: 49. 101.0KiB allocated for chunks. 101.0KiB in use in bin. 99.4KiB client-requested in use in bin.
2020-09-15 16:28:22.603977: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): Total Chunks: 12, Chunks in use: 12. 48.0KiB allocated for chunks. 48.0KiB in use in bin. 48.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604033: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604092: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.604165: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): Total Chunks: 3, Chunks in use: 2. 102.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604297: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): Total Chunks: 2, Chunks in use: 2. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604367: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): Total Chunks: 9, Chunks in use: 8. 1.14MiB allocated for chunks. 1.00MiB in use in bin. 1.00MiB client-requested in use in bin.
2020-09-15 16:28:22.604435: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): Total Chunks: 3, Chunks in use: 3. 864.0KiB allocated for chunks. 864.0KiB in use in bin. 864.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604496: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): Total Chunks: 10, Chunks in use: 8. 5.28MiB allocated for chunks. 4.28MiB in use in bin. 4.00MiB client-requested in use in bin.
2020-09-15 16:28:22.604655: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): Total Chunks: 9, Chunks in use: 9. 10.88MiB allocated for chunks. 10.88MiB in use in bin. 10.12MiB client-requested in use in bin.
2020-09-15 16:28:22.604720: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): Total Chunks: 5, Chunks in use: 3. 11.00MiB allocated for chunks. 6.50MiB in use in bin. 6.00MiB client-requested in use in bin.
2020-09-15 16:28:22.604786: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): Total Chunks: 10, Chunks in use: 9. 46.88MiB allocated for chunks. 42.38MiB in use in bin. 40.50MiB client-requested in use in bin.
2020-09-15 16:28:22.604846: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.604909: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): Total Chunks: 4, Chunks in use: 4. 74.35MiB allocated for chunks. 74.35MiB in use in bin. 72.00MiB client-requested in use in bin.
2020-09-15 16:28:22.604972: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): Total Chunks: 1, Chunks in use: 1. 32.50MiB allocated for chunks. 32.50MiB in use in bin. 18.00MiB client-requested in use in bin.
2020-09-15 16:28:22.605121: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.605181: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.605238: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.605298: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 18.00MiB was 16.00MiB, Chunk State:
2020-09-15 16:28:22.605339: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1048576
2020-09-15 16:28:22.647695: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0000 of size 1280 next 1
2020-09-15 16:28:22.647782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0500 of size 256 next 2
2020-09-15 16:28:22.647942: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0600 of size 256 next 5
2020-09-15 16:28:22.647989: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0700 of size 256 next 4
2020-09-15 16:28:22.648030: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0800 of size 256 next 7
2020-09-15 16:28:22.648073: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0900 of size 256 next 8
2020-09-15 16:28:22.648110: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0a00 of size 256 next 9
2020-09-15 16:28:22.648155: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0b00 of size 256 next 11
2020-09-15 16:28:22.648260: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0c00 of size 256 next 13
2020-09-15 16:28:22.648326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0d00 of size 256 next 14
2020-09-15 16:28:22.648377: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0e00 of size 256 next 15
2020-09-15 16:28:22.648426: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0f00 of size 256 next 17
2020-09-15 16:28:22.648589: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1000 of size 256 next 19
2020-09-15 16:28:22.648640: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1100 of size 256 next 20
2020-09-15 16:28:22.648681: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1200 of size 256 next 22
2020-09-15 16:28:22.648734: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1300 of size 256 next 24
2020-09-15 16:28:22.648776: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1400 of size 256 next 25
2020-09-15 16:28:22.648820: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1500 of size 256 next 26
2020-09-15 16:28:22.648865: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1600 of size 512 next 27
2020-09-15 16:28:22.648910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1800 of size 512 next 30
2020-09-15 16:28:22.648954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1a00 of size 512 next 31
2020-09-15 16:28:22.648993: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1c00 of size 512 next 32
2020-09-15 16:28:22.649031: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1e00 of size 256 next 33
2020-09-15 16:28:22.649068: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1f00 of size 256 next 35
2020-09-15 16:28:22.649225: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2000 of size 256 next 36
2020-09-15 16:28:22.649269: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2100 of size 256 next 37
2020-09-15 16:28:22.649308: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2200 of size 512 next 3
2020-09-15 16:28:22.649349: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2400 of size 3584 next 6
2020-09-15 16:28:22.649388: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3200 of size 512 next 38
2020-09-15 16:28:22.649428: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3400 of size 512 next 40
2020-09-15 16:28:22.649515: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3600 of size 512 next 41
2020-09-15 16:28:22.649556: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3800 of size 256 next 42
2020-09-15 16:28:22.649601: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3900 of size 256 next 44
2020-09-15 16:28:22.649640: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3a00 of size 256 next 45
2020-09-15 16:28:22.649691: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3b00 of size 256 next 46
2020-09-15 16:28:22.649728: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3c00 of size 512 next 47
2020-09-15 16:28:22.649770: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3e00 of size 512 next 49
2020-09-15 16:28:22.649811: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4000 of size 512 next 50
2020-09-15 16:28:22.649857: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4200 of size 512 next 51
2020-09-15 16:28:22.649906: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4400 of size 1024 next 53
2020-09-15 16:28:22.649947: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4800 of size 1024 next 56
2020-09-15 16:28:22.649992: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4c00 of size 1024 next 57
2020-09-15 16:28:22.650032: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5000 of size 1024 next 58
2020-09-15 16:28:22.650076: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5400 of size 512 next 59
2020-09-15 16:28:22.650115: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5600 of size 512 next 61
2020-09-15 16:28:22.650165: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5800 of size 512 next 63
2020-09-15 16:28:22.650308: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5a00 of size 512 next 64
2020-09-15 16:28:22.650350: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5c00 of size 1024 next 65
2020-09-15 16:28:22.650392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6000 of size 1024 next 66
2020-09-15 16:28:22.650430: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6400 of size 1024 next 67
2020-09-15 16:28:22.651437: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6800 of size 1024 next 68
2020-09-15 16:28:22.651528: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6c00 of size 512 next 69
2020-09-15 16:28:22.651578: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6e00 of size 512 next 71
2020-09-15 16:28:22.651621: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7000 of size 512 next 18
2020-09-15 16:28:22.651668: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7200 of size 8192 next 16
2020-09-15 16:28:22.651713: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f9200 of size 32768 next 34
2020-09-15 16:28:22.651761: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00901200 of size 32768 next 43
2020-09-15 16:28:22.651805: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909200 of size 512 next 72
2020-09-15 16:28:22.651847: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909400 of size 1024 next 75
2020-09-15 16:28:22.651894: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909800 of size 1024 next 76
2020-09-15 16:28:22.651943: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909c00 of size 1024 next 77
2020-09-15 16:28:22.651986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a000 of size 1024 next 78
2020-09-15 16:28:22.652030: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a400 of size 512 next 79
2020-09-15 16:28:22.652070: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a600 of size 512 next 80
2020-09-15 16:28:22.652311: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a800 of size 512 next 82
2020-09-15 16:28:22.652408: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090aa00 of size 512 next 83
2020-09-15 16:28:22.652459: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ac00 of size 1024 next 85
2020-09-15 16:28:22.652500: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b000 of size 1024 next 86
2020-09-15 16:28:22.652543: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b400 of size 1024 next 87
2020-09-15 16:28:22.652589: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b800 of size 1024 next 88
2020-09-15 16:28:22.652870: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090bc00 of size 512 next 90
2020-09-15 16:28:22.652933: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090be00 of size 512 next 91
2020-09-15 16:28:22.652982: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c000 of size 512 next 92
2020-09-15 16:28:22.653025: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c200 of size 512 next 93
2020-09-15 16:28:22.653055: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c400 of size 1024 next 95
2020-09-15 16:28:22.653081: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c800 of size 1024 next 97
2020-09-15 16:28:22.653107: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090cc00 of size 1024 next 98
2020-09-15 16:28:22.653130: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d000 of size 1024 next 99
2020-09-15 16:28:22.653160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d400 of size 512 next 101
2020-09-15 16:28:22.653842: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d600 of size 512 next 102
2020-09-15 16:28:22.657624: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d800 of size 512 next 103
2020-09-15 16:28:22.657723: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090da00 of size 512 next 104
2020-09-15 16:28:22.657783: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090dc00 of size 1024 next 106
2020-09-15 16:28:22.657836: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e000 of size 1024 next 107
2020-09-15 16:28:22.657896: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e400 of size 1024 next 108
2020-09-15 16:28:22.657941: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e800 of size 1024 next 109
2020-09-15 16:28:22.657984: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ec00 of size 512 next 111
2020-09-15 16:28:22.658032: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ee00 of size 512 next 112
2020-09-15 16:28:22.658089: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f000 of size 512 next 113
2020-09-15 16:28:22.658132: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f200 of size 512 next 114
2020-09-15 16:28:22.658174: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f400 of size 1024 next 116
2020-09-15 16:28:22.658213: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f800 of size 1024 next 117
2020-09-15 16:28:22.658256: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090fc00 of size 1024 next 118
2020-09-15 16:28:22.658295: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910000 of size 1024 next 119
2020-09-15 16:28:22.658334: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910400 of size 512 next 121
2020-09-15 16:28:22.658373: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910600 of size 512 next 122
2020-09-15 16:28:22.658411: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910800 of size 512 next 123
2020-09-15 16:28:22.658449: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910a00 of size 512 next 124
2020-09-15 16:28:22.658489: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910c00 of size 1024 next 125
2020-09-15 16:28:22.658527: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911000 of size 1024 next 126
2020-09-15 16:28:22.658567: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911400 of size 1024 next 127
2020-09-15 16:28:22.658605: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911800 of size 1024 next 128
2020-09-15 16:28:22.658644: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911c00 of size 512 next 130
2020-09-15 16:28:22.658682: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911e00 of size 512 next 131
2020-09-15 16:28:22.658720: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912000 of size 512 next 132
2020-09-15 16:28:22.658758: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912200 of size 512 next 133
2020-09-15 16:28:22.658798: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912400 of size 1024 next 136
2020-09-15 16:28:22.658836: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912800 of size 1024 next 137
2020-09-15 16:28:22.658874: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912c00 of size 1024 next 138
2020-09-15 16:28:22.658913: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913000 of size 1024 next 139
2020-09-15 16:28:22.658957: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913400 of size 2048 next 140
2020-09-15 16:28:22.659005: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913c00 of size 2048 next 141
2020-09-15 16:28:22.659055: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914400 of size 2048 next 142
2020-09-15 16:28:22.659102: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914c00 of size 2048 next 144
2020-09-15 16:28:22.659150: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915400 of size 1024 next 145
2020-09-15 16:28:22.659197: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915800 of size 1024 next 146
2020-09-15 16:28:22.659246: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915c00 of size 1024 next 148
2020-09-15 16:28:22.659301: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916000 of size 1024 next 149
2020-09-15 16:28:22.659351: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916400 of size 3584 next 12
2020-09-15 16:28:22.659394: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00917200 of size 73728 next 10
2020-09-15 16:28:22.659445: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929200 of size 2048 next 150
2020-09-15 16:28:22.659698: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929a00 of size 2048 next 153
2020-09-15 16:28:22.659763: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092a200 of size 2048 next 154
2020-09-15 16:28:22.659810: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092aa00 of size 1024 next 156
2020-09-15 16:28:22.659851: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ae00 of size 1024 next 157
2020-09-15 16:28:22.659891: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b200 of size 1024 next 158
2020-09-15 16:28:22.659937: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b600 of size 1024 next 159
2020-09-15 16:28:22.659981: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ba00 of size 2048 next 160
2020-09-15 16:28:22.660023: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092c200 of size 2048 next 162
2020-09-15 16:28:22.660065: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ca00 of size 2048 next 163
2020-09-15 16:28:22.660107: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092d200 of size 2048 next 164
2020-09-15 16:28:22.660151: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092da00 of size 1024 next 165
2020-09-15 16:28:22.660233: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092de00 of size 1024 next 166
2020-09-15 16:28:22.660283: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e200 of size 1024 next 167
2020-09-15 16:28:22.660327: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e600 of size 1024 next 168
2020-09-15 16:28:22.660374: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ea00 of size 2048 next 169
2020-09-15 16:28:22.660419: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092f200 of size 2048 next 171
2020-09-15 16:28:22.660462: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092fa00 of size 2048 next 172
2020-09-15 16:28:22.660502: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930200 of size 2048 next 173
2020-09-15 16:28:22.660540: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930a00 of size 1024 next 175
2020-09-15 16:28:22.660581: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930e00 of size 1024 next 176
2020-09-15 16:28:22.660622: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931200 of size 1024 next 177
2020-09-15 16:28:22.660661: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931600 of size 1024 next 178
2020-09-15 16:28:22.660699: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931a00 of size 2048 next 180
2020-09-15 16:28:22.660738: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932200 of size 2048 next 182
2020-09-15 16:28:22.660787: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932a00 of size 2048 next 183
2020-09-15 16:28:22.660841: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933200 of size 2048 next 184
2020-09-15 16:28:22.660885: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933a00 of size 1024 next 186
2020-09-15 16:28:22.660929: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933e00 of size 1024 next 187
2020-09-15 16:28:22.661114: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934200 of size 1024 next 188
2020-09-15 16:28:22.661160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934600 of size 1024 next 189
2020-09-15 16:28:22.661207: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934a00 of size 2048 next 190
2020-09-15 16:28:22.661248: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935200 of size 2048 next 191
2020-09-15 16:28:22.661290: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935a00 of size 2048 next 193
2020-09-15 16:28:22.661332: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936200 of size 2048 next 194
2020-09-15 16:28:22.661377: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936a00 of size 1024 next 196
2020-09-15 16:28:22.661420: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936e00 of size 1024 next 197
2020-09-15 16:28:22.661467: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937200 of size 1024 next 198
2020-09-15 16:28:22.661512: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937600 of size 1024 next 199
2020-09-15 16:28:22.661561: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937a00 of size 2048 next 202
2020-09-15 16:28:22.661610: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938200 of size 2048 next 203
2020-09-15 16:28:22.661655: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938a00 of size 2048 next 204
2020-09-15 16:28:22.661695: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939200 of size 2048 next 205
2020-09-15 16:28:22.661756: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939a00 of size 1024 next 207
2020-09-15 16:28:22.661805: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939e00 of size 1024 next 208
2020-09-15 16:28:22.661849: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a200 of size 1024 next 209
2020-09-15 16:28:22.661897: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a600 of size 1024 next 210
2020-09-15 16:28:22.661944: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093aa00 of size 2048 next 23
2020-09-15 16:28:22.662090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093b200 of size 73728 next 21
2020-09-15 16:28:22.662145: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094d200 of size 2048 next 212
2020-09-15 16:28:22.662195: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094da00 of size 2048 next 213
2020-09-15 16:28:22.662245: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094e200 of size 2048 next 214
2020-09-15 16:28:22.662296: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ea00 of size 1024 next 216
2020-09-15 16:28:22.662346: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ee00 of size 1024 next 217
2020-09-15 16:28:22.662392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f200 of size 1024 next 218
2020-09-15 16:28:22.662440: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f600 of size 1024 next 219
2020-09-15 16:28:22.662486: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094fa00 of size 2048 next 221
2020-09-15 16:28:22.662530: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950200 of size 2048 next 222
2020-09-15 16:28:22.662573: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950a00 of size 2048 next 223
2020-09-15 16:28:22.662721: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951200 of size 2048 next 224
2020-09-15 16:28:22.662773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951a00 of size 4096 next 225
2020-09-15 16:28:22.662818: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00952a00 of size 4096 next 228
2020-09-15 16:28:22.662861: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00953a00 of size 4096 next 229
2020-09-15 16:28:22.662908: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00954a00 of size 4096 next 230
2020-09-15 16:28:22.662952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00955a00 of size 2048 next 231
2020-09-15 16:28:22.662999: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956200 of size 2048 next 232
2020-09-15 16:28:22.663046: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956a00 of size 2048 next 234
2020-09-15 16:28:22.663093: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957200 of size 2048 next 235
2020-09-15 16:28:22.663141: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957a00 of size 4096 next 236
2020-09-15 16:28:22.663185: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00958a00 of size 4096 next 238
2020-09-15 16:28:22.663226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00959a00 of size 4096 next 239
2020-09-15 16:28:22.663267: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095aa00 of size 4096 next 240
2020-09-15 16:28:22.663406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ba00 of size 2048 next 242
2020-09-15 16:28:22.663517: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095c200 of size 2048 next 243
2020-09-15 16:28:22.664124: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ca00 of size 2048 next 244
2020-09-15 16:28:22.664229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095d200 of size 2048 next 245
2020-09-15 16:28:22.664280: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095da00 of size 4096 next 246
2020-09-15 16:28:22.664324: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ea00 of size 4096 next 247
2020-09-15 16:28:22.664365: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095fa00 of size 4096 next 249
2020-09-15 16:28:22.664406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00960a00 of size 4096 next 250
2020-09-15 16:28:22.664445: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00961a00 of size 2048 next 252
2020-09-15 16:28:22.664801: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962200 of size 2048 next 253
2020-09-15 16:28:22.664860: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962a00 of size 2048 next 254
2020-09-15 16:28:22.664901: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00963200 of size 2048 next 255
2020-09-15 16:28:22.666022: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f00963a00 of size 38912 next 81
2020-09-15 16:28:22.666098: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096d200 of size 131072 next 62
2020-09-15 16:28:22.666143: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0098d200 of size 131072 next 60
2020-09-15 16:28:22.666186: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009ad200 of size 131072 next 70
2020-09-15 16:28:22.666227: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f009cd200 of size 142848 next 18446744073709551615
2020-09-15 16:28:22.666269: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 2097152
2020-09-15 16:28:22.666313: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009f0000 of size 294912 next 29
2020-09-15 16:28:22.666354: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a38000 of size 294912 next 39
2020-09-15 16:28:22.666392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a80000 of size 294912 next 48
2020-09-15 16:28:22.666431: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ac8000 of size 524288 next 155
2020-09-15 16:28:22.666470: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00b48000 of size 688128 next 18446744073709551615
2020-09-15 16:28:22.666507: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 4194304
2020-09-15 16:28:22.666545: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00bf0000 of size 131072 next 89
2020-09-15 16:28:22.666584: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c10000 of size 131072 next 100
2020-09-15 16:28:22.666622: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c30000 of size 131072 next 110
2020-09-15 16:28:22.666663: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c50000 of size 131072 next 120
2020-09-15 16:28:22.666703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c70000 of size 131072 next 129
2020-09-15 16:28:22.666744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f00c90000 of size 524288 next 55
2020-09-15 16:28:22.666785: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00d10000 of size 1179648 next 52
2020-09-15 16:28:22.666830: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00e30000 of size 1835008 next 18446744073709551615
2020-09-15 16:28:22.666876: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 8388608
2020-09-15 16:28:22.666921: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ff0000 of size 1179648 next 74
2020-09-15 16:28:22.666969: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01110000 of size 1179648 next 84
2020-09-15 16:28:22.667018: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f01230000 of size 524288 next 147
2020-09-15 16:28:22.667060: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f012b0000 of size 655360 next 96
2020-09-15 16:28:22.667510: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01350000 of size 1179648 next 94
2020-09-15 16:28:22.667567: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01470000 of size 1179648 next 105
2020-09-15 16:28:22.667607: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01590000 of size 1179648 next 115
2020-09-15 16:28:22.667646: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f016b0000 of size 1310720 next 18446744073709551615
2020-09-15 16:28:22.667687: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 16777216
2020-09-15 16:28:22.667727: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f017f0000 of size 1179648 next 135
2020-09-15 16:28:22.667766: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01910000 of size 524288 next 174
2020-09-15 16:28:22.667806: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01990000 of size 2097152 next 241
2020-09-15 16:28:22.669192: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01b90000 of size 2097152 next 251
2020-09-15 16:28:22.669273: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f01d90000 of size 4718592 next 143
2020-09-15 16:28:22.669322: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02210000 of size 6160384 next 18446744073709551615
2020-09-15 16:28:22.669366: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 33554432
2020-09-15 16:28:22.669411: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f027f0000 of size 4718592 next 152
2020-09-15 16:28:22.669454: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02c70000 of size 4718592 next 161
2020-09-15 16:28:22.669494: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f030f0000 of size 4718592 next 170
2020-09-15 16:28:22.669533: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03570000 of size 524288 next 185
2020-09-15 16:28:22.669573: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f035f0000 of size 524288 next 195
2020-09-15 16:28:22.669614: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03670000 of size 524288 next 206
2020-09-15 16:28:22.669658: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f036f0000 of size 524288 next 215
2020-09-15 16:28:22.669700: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f03770000 of size 2621440 next 181
2020-09-15 16:28:22.669744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f039f0000 of size 4718592 next 179
2020-09-15 16:28:22.669790: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f03e70000 of size 2097152 next 233
2020-09-15 16:28:22.669831: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04070000 of size 2621440 next 192
2020-09-15 16:28:22.669872: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f042f0000 of size 5242880 next 18446744073709551615
2020-09-15 16:28:22.669913: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 67108864
2020-09-15 16:28:22.669955: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f047f0000 of size 4718592 next 201
2020-09-15 16:28:22.669996: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04c70000 of size 4718592 next 211
2020-09-15 16:28:22.670131: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f050f0000 of size 4718592 next 220
2020-09-15 16:28:22.670174: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f05570000 of size 18874368 next 237
2020-09-15 16:28:22.670213: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f06770000 of size 34078720 next 18446744073709551615
2020-09-15 16:28:22.670249: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 59088896
2020-09-15 16:28:22.670290: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f087f0000 of size 18874368 next 227
2020-09-15 16:28:22.670328: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f099f0000 of size 18874368 next 248
2020-09-15 16:28:22.670366: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0abf0000 of size 21340160 next 18446744073709551615
2020-09-15 16:28:22.670401: I tensorflow/core/common_runtime/bfc_allocator.cc:955] Summary of in-use Chunks by size:
2020-09-15 16:28:22.670449: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 25 Chunks of size 256 totalling 6.2KiB
2020-09-15 16:28:22.670494: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 44 Chunks of size 512 totalling 22.0KiB
2020-09-15 16:28:22.670537: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 68 Chunks of size 1024 totalling 68.0KiB
2020-09-15 16:28:22.670577: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1280 totalling 1.2KiB
2020-09-15 16:28:22.670619: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 47 Chunks of size 2048 totalling 94.0KiB
2020-09-15 16:28:22.670798: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 3584 totalling 7.0KiB
2020-09-15 16:28:22.671463: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 12 Chunks of size 4096 totalling 48.0KiB
2020-09-15 16:28:22.671519: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 8192 totalling 8.0KiB
2020-09-15 16:28:22.671553: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 32768 totalling 64.0KiB
2020-09-15 16:28:22.671587: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 73728 totalling 144.0KiB
2020-09-15 16:28:22.671621: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 8 Chunks of size 131072 totalling 1.00MiB
2020-09-15 16:28:22.671654: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 294912 totalling 864.0KiB
2020-09-15 16:28:22.671686: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 6 Chunks of size 524288 totalling 3.00MiB
2020-09-15 16:28:22.671723: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 655360 totalling 640.0KiB
2020-09-15 16:28:22.671762: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 688128 totalling 672.0KiB
2020-09-15 16:28:22.671795: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 1179648 totalling 7.88MiB
2020-09-15 16:28:22.671828: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1310720 totalling 1.25MiB
2020-09-15 16:28:22.671862: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1835008 totalling 1.75MiB
2020-09-15 16:28:22.671895: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 2097152 totalling 4.00MiB
2020-09-15 16:28:22.671924: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 2621440 totalling 2.50MiB
2020-09-15 16:28:22.671953: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 4718592 totalling 31.50MiB
2020-09-15 16:28:22.671982: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 5242880 totalling 5.00MiB
2020-09-15 16:28:22.672012: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 6160384 totalling 5.88MiB
2020-09-15 16:28:22.672040: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 18874368 totalling 54.00MiB
2020-09-15 16:28:22.672068: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 21340160 totalling 20.35MiB
2020-09-15 16:28:22.672097: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 34078720 totalling 32.50MiB
2020-09-15 16:28:22.672126: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 173.18MiB
2020-09-15 16:28:22.672152: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocatedbytes: 192258048 memorylimit: 192258048 available bytes: 0 curr_region_allocationbytes: 268435456
2020-09-15 16:28:22.672265: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 192258048
InUse: 181590528
MaxInUse: 181590528
NumAllocs: 1232
MaxAllocSize: 34078720
2020-09-15 16:28:22.672343: W tensorflow/core/common_runtime/bfcallocator.cc:429] *******xxxxxxx**x
2020-09-15 16:28:22.690033: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "object_tracker.py", line 163, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "object_tracker.py", line 52, in main
yolo = YoloV3(classes=FLAGS.num_classes)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 211, in YoloV3
x_36, x_61, x = Darknet(name='yolo_darknet')(x)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 78, in Darknet
x = DarknetBlock(x, 1024, 4)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 67, in DarknetBlock
x = DarknetResidual(x, filters)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 59, in DarknetResidual
x = DarknetConv(x, filters, 3)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 49, in DarknetConv
use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 748, in call
self._maybe_build(inputs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2116, in _maybe_build
self.build(input_shapes)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py", line 161, in build
dtype=self.dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 446, in add_weight
caching_device=caching_device)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
*kwargs_for_getter)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 142, in make_variable
shape=variable_shape if variable_shape else None)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 258, in call
return cls._variable_v1_call(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 197, in
previous_getter = lambda kwargs: default_variable_creator(None, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py", line 2596, in default_variable_creator
shape=shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 262, in call
return super(VariableMetaclass, cls).call(*args, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1411, in init
distribute_strategy=distribute_strategy)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1542, in _init_from_args
initial_value() if init_from_fn else initial_value,
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 122, in
init_val = lambda: initializer(shape, dtype=dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 425, in call
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 788, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py", line 273, in random_uniform
result = math_ops.add(rnd (maxval - minval), minval, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 343, in add
_ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_49/kernel/Initializer/random_uniform/
2020-09-15 16:28:22.672126: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 173.18MiB
2020-09-15 16:28:22.672152: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocatedbytes: 192258048 memorylimit: 192258048 available bytes: 0 curr_region_allocationbytes: 268435456
2020-09-15 16:28:22.672265: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 192258048
InUse: 181590528
MaxInUse: 181590528
NumAllocs: 1232
MaxAllocSize: 34078720
2020-09-15 16:28:22.672343: W tensorflow/core/common_runtime/bfcallocator.cc:429] *******xxxxxxx**x
2020-09-15 16:28:22.690033: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "object_tracker.py", line 163, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "object_tracker.py", line 52, in main
yolo = YoloV3(classes=FLAGS.num_classes)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 211, in YoloV3
x_36, x_61, x = Darknet(name='yolo_darknet')(x)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 78, in Darknet
x = DarknetBlock(x, 1024, 4)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 67, in DarknetBlock
x = DarknetResidual(x, filters)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 59, in DarknetResidual
x = DarknetConv(x, filters, 3)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 49, in DarknetConv
use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 748, in call
self._maybe_build(inputs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2116, in _maybe_build
self.build(input_shapes)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py", line 161, in build
dtype=self.dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 446, in add_weight
caching_device=caching_device)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
*kwargs_for_getter)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 142, in make_variable
shape=variable_shape if variable_shape else None)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 258, in call
return cls._variable_v1_call(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 197, in
previous_getter = lambda kwargs: default_variable_creator(None, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py", line 2596, in default_variable_creator
shape=shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 262, in call
return super(VariableMetaclass, cls).call(*args, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1411, in init
distribute_strategy=distribute_strategy)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1542, in _init_from_args
initial_value() if init_from_fn else initial_value,
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 122, in
init_val = lambda: initializer(shape, dtype=dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 425, in call
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 788, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py", line 273, in random_uniform
result = math_ops.add(rnd (maxval - minval), minval, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 343, in add
_ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_49/kernel/Initializer/random_uniform/
Hi,
when I tried to run the object tracker with
python object_tracker.py --video 0 --output ./data/video/results.aviunfortunatelly an error occurs. You can find the whole output below.
Here are some of the errors which are listed within the complete output below:
I'm using a Jetson Nano with JetPack 4.3 and Tensorflow version 2.1.0.
I would appreciate some good advice. BR
Complete output text