I ran the script ./train unseen_speakers
and trained it for train(run_name, 0,1000, 3, 100, 50, 75, 32, 4) (1000epoch and four videos)
but when I ran for 145 epochs,got this problem:
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 19 Chunks of size 786432 totalling 14.25MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 12 Chunks of size 1572864 totalling 18.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1720320 totalling 1.64MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 2073600 totalling 1.98MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 2365440 totalling 2.26MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 13 Chunks of size 5308416 totalling 65.81MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 5529600 totalling 5.27MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 6099712 totalling 5.82MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 8294400 totalling 23.73MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 11520000 totalling 10.99MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 13630464 totalling 13.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 14745600 totalling 14.06MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 17508608 totalling 16.70MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 20756736 totalling 19.79MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 23040000 totalling 109.86MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 36476416 totalling 34.79MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 48000000 totalling 228.88MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.38GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit: 1504051200
InUse: 1478468352
MaxInUse: 1504051200
NumAllocs: 91657676
MaxAllocSize: 509946624
W tensorflow/core/common_runtime/bfc_allocator.cc:274] ****
W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 7.91MiB. See logs for memory state.
I use geforce gtx960,is this problem about my gpu'memory?
Dose anyone happen this problem?
or can anyone tell me which gpu do you use,
and run for how many epochs and train for how many videos
thanks very much!!
liviust
commented
6 years ago
I ran the script ./train unseen_speakers and trained it for train(run_name, 0,1000, 3, 100, 50, 75, 32, 4) (1000epoch and four videos) but when I ran for 145 epochs,got this problem:
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 19 Chunks of size 786432 totalling 14.25MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 12 Chunks of size 1572864 totalling 18.00MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1720320 totalling 1.64MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 2073600 totalling 1.98MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 2365440 totalling 2.26MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 13 Chunks of size 5308416 totalling 65.81MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 5529600 totalling 5.27MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 6099712 totalling 5.82MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 8294400 totalling 23.73MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 11520000 totalling 10.99MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 13630464 totalling 13.00MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 14745600 totalling 14.06MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 17508608 totalling 16.70MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 20756736 totalling 19.79MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 23040000 totalling 109.86MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 36476416 totalling 34.79MiB I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 48000000 totalling 228.88MiB I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.38GiB I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: Limit: 1504051200 InUse: 1478468352 MaxInUse: 1504051200 NumAllocs: 91657676 MaxAllocSize: 509946624
W tensorflow/core/common_runtime/bfc_allocator.cc:274] **** W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 7.91MiB. See logs for memory state.
I use geforce gtx960,is this problem about my gpu'memory? Dose anyone happen this problem? or can anyone tell me which gpu do you use, and run for how many epochs and train for how many videos thanks very much!!