2023-06-30 12:08:16.805705: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 171305984 totalling 163.37MiB
2023-06-30 12:08:16.808694: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 184549376 totalling 176.00MiB
2023-06-30 12:08:16.809081: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 262144000 totalling 250.00MiB
2023-06-30 12:08:16.809482: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 335544320 totalling 320.00MiB
2023-06-30 12:08:16.809970: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 394264576 totalling 376.00MiB
2023-06-30 12:08:16.810709: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 4.44GiB
2023-06-30 12:08:16.811683: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocatedbytes: 4979687424 memorylimit: 4979687424 available bytes: 0 curr_region_allocationbytes: 9959374848
2023-06-30 12:08:16.813068: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 4979687424
InUse: 4764532736
MaxInUse: 4884267008
NumAllocs: 495
MaxAllocSize: 1526726656
2023-06-30 12:08:16.813297: W tensorflow/core/common_runtime/bfcallocator.cc:429] ****x*****x
2023-06-30 12:08:16.814366: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at conv_ops.cc:539 : Resource exhausted: OOM when allocating tensor with shape[20,256,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1367, in _do_call
return fn(*args)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1352, in _run_fn
target_list, run_metadata)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1445, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[20,256,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node Conv2D_37}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "style.py", line 167, in
main()
File "style.py", line 147, in main
for preds, losses, i, epoch in optimize(*args, **kwargs):
File "src\optimize.py", line 114, in optimize
train_step.run(feed_dict=feed_dict)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\ops.py", line 2391, in run
_run_using_default_session(self, feed_dict, self.graph, session)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\ops.py", line 5347, in _run_using_default_session
session.run(operation, feed_dict)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 960, in run
run_metadata_ptr)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1183, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1361, in _do_run
run_metadata)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1386, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[20,256,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node Conv2D_37 (defined at src\vgg.py:54) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Errors may have originated from an input operation.
Input Source operations connected to node Conv2D_37:
Const_23 (defined at src\vgg.py:53)
Relu_32 (defined at src\vgg.py:43)
Original stack trace for 'Conv2D_37':
File "style.py", line 167, in
main()
File "style.py", line 147, in main
for preds, losses, i, epoch in optimize(*args, **kwargs):
File "src\optimize.py", line 60, in optimize
net = vgg.net(vgg_path, preds_pre)
File "src\vgg.py", line 41, in net
current = _conv_layer(current, kernels, bias)
File "src\vgg.py", line 54, in _conv_layer
padding='SAME')
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 1914, in conv2d_v2
name=name)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 2011, in conv2d
name=name)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 969, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 742, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3322, in _create_op_internal
op_def=op_def)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1756, in init
self._traceback = tf_stack.extract_stack()
2023-06-30 12:08:16.805705: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 171305984 totalling 163.37MiB 2023-06-30 12:08:16.808694: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 184549376 totalling 176.00MiB 2023-06-30 12:08:16.809081: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 262144000 totalling 250.00MiB 2023-06-30 12:08:16.809482: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 335544320 totalling 320.00MiB 2023-06-30 12:08:16.809970: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 394264576 totalling 376.00MiB 2023-06-30 12:08:16.810709: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 4.44GiB 2023-06-30 12:08:16.811683: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocatedbytes: 4979687424 memorylimit: 4979687424 available bytes: 0 curr_region_allocationbytes: 9959374848 2023-06-30 12:08:16.813068: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats: Limit: 4979687424 InUse: 4764532736 MaxInUse: 4884267008 NumAllocs: 495 MaxAllocSize: 1526726656
2023-06-30 12:08:16.813297: W tensorflow/core/common_runtime/bfcallocator.cc:429] ****x*****x 2023-06-30 12:08:16.814366: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at conv_ops.cc:539 : Resource exhausted: OOM when allocating tensor with shape[20,256,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1367, in _do_call return fn(*args) File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1352, in _run_fn target_list, run_metadata) File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1445, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[20,256,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node Conv2D_37}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "style.py", line 167, in
main()
File "style.py", line 147, in main
for preds, losses, i, epoch in optimize(*args, **kwargs):
File "src\optimize.py", line 114, in optimize
train_step.run(feed_dict=feed_dict)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\ops.py", line 2391, in run
_run_using_default_session(self, feed_dict, self.graph, session)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\ops.py", line 5347, in _run_using_default_session
session.run(operation, feed_dict)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 960, in run
run_metadata_ptr)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1183, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1361, in _do_run
run_metadata)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1386, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[20,256,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node Conv2D_37 (defined at src\vgg.py:54) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Errors may have originated from an input operation. Input Source operations connected to node Conv2D_37: Const_23 (defined at src\vgg.py:53) Relu_32 (defined at src\vgg.py:43)
Original stack trace for 'Conv2D_37': File "style.py", line 167, in
main()
File "style.py", line 147, in main
for preds, losses, i, epoch in optimize(*args, **kwargs):
File "src\optimize.py", line 60, in optimize
net = vgg.net(vgg_path, preds_pre)
File "src\vgg.py", line 41, in net
current = _conv_layer(current, kernels, bias)
File "src\vgg.py", line 54, in _conv_layer
padding='SAME')
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 1914, in conv2d_v2
name=name)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 2011, in conv2d
name=name)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 969, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 742, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3322, in _create_op_internal
op_def=op_def)
File "C:\Users\qq978.conda\envs\tf-gpu\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1756, in init
self._traceback = tf_stack.extract_stack()