A tensorflow implementation of "Fast and Accurate Image Super Resolution by Deep CNN with Skip Connection and Network in Network", a deep learning based Single-Image Super-Resolution (SISR) model.
I've tried 10x, 5x, 2x and i'm receiving the same error.
the image is 7995x4586
It also doesn't seem to be using the GPU, I'm not sure if it's designed to or not with the sr.py
These are the call i am using
python train.py --dataset=hr384 --layers=8 --filters=96 --training_images=100 --scale 2
python sr.py --file=SatImageSm.png --layers=8 --filters=96 --training_images=100 --scale 2
DCSCN v2-------------------------------------
2021/7/13 13:35:48 [dcscn_L8_F96to48_NIN_A64_PS_R1F32]
Feature:96 82 75 68 62 57 52 48 Total: (540)64 32 32 384 1 Complexity:687,268 Receptive Fields:21
CNN1/conv_W:0 (3, 3, 1, 96)=864, CNN1/conv_B:0 96, CNN1/prelu/CNN1_prelu:0 96, CNN2/conv_W:0 (3, 3, 96, 82)=70848, CNN2/conv_B:0 82, CNN2/prelu/CNN2_prelu:0 82, CNN3/conv_W:0 (3, 3, 82, 75)=55350, CNN3/conv_B:0 75, CNN3/prelu/CNN3_prelu:0 75, CNN4/conv_W:0 (3, 3, 75, 68)=45900, CNN4/conv_B:0 68, CNN4/prelu/CNN4_prelu:0 68, CNN5/conv_W:0 (3, 3, 68, 62)=37944, CNN5/conv_B:0 62, CNN5/prelu/CNN5_prelu:0 62, CNN6/conv_W:0 (3, 3, 62, 57)=31806, CNN6/conv_B:0 57, CNN6/prelu/CNN6_prelu:0 57, CNN7/conv_W:0 (3, 3, 57, 52)=26676, CNN7/conv_B:0 52, CNN7/prelu/CNN7_prelu:0 52, CNN8/conv_W:0 (3, 3, 52, 48)=22464, CNN8/conv_B:0 48, CNN8/prelu/CNN8_prelu:0 48, A1/conv_W:0 (1, 1, 540, 64)=34560, A1/conv_B:0 64, A1/prelu/A1_prelu:0 64, B1/conv_W:0 (1, 1, 540, 32)=17280, B1/conv_B:0 32, B1/prelu/B1_prelu:0 32, B2/conv_W:0 (3, 3, 32, 32)=9216, B2/conv_B:0 32, B2/prelu/B2_prelu:0 32, Up-PS/Up-PS_CNN/conv_W:0 (3, 3, 96, 384)=331776, Up-PS/Up-PS_CNN/conv_B:0 384, R-CNN1/conv_W:0 (3, 3, 96, 1)=864,
Total 36 variables, 687,268 params
Model initialized.
Model restored [ models/dcscn_L8_F96to48_NIN_A64_PS_R1F32.ckpt ].
Loaded [SatImageSm.png]: 7995 x 4586 x 3
Saved [output/dcscn_L8_F96to48_NIN_A64_PS_R1F32/SatImageSm.png]
Saved [output/dcscn_L8_F96to48_NIN_A64_PS_R1F32/SatImageSm_bicubic.png]
Saved [output/dcscn_L8_F96to48_NIN_A64_PS_R1F32/SatImageSm_bicubic_y.png]
2021-07-13 13:39:01.532171: W tensorflow/core/common_runtime/bfc_allocator.cc:456] Allocator (GPU_0_bfc) ran out of memory trying to allocate 13.11GiB (rounded to 14079386880)requested by op CNN1/CNN1_conv
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2021-07-13 13:39:01.532246: I tensorflow/core/common_runtime/bfc_allocator.cc:991] BFCAllocator dump for GPU_0_bfc
2021-07-13 13:39:01.532704: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (256): Total Chunks: 87, Chunks in use: 87. 21.8KiB allocated for chunks. 21.8KiB in use in bin. 13.4KiB client-requested in use in bin.
2021-07-13 13:39:01.532734: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 12.5KiB client-requested in use in bin.
2021-07-13 13:39:01.532763: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (1024): Total Chunks: 6, Chunks in use: 6. 8.8KiB allocated for chunks. 8.8KiB in use in bin. 8.5KiB client-requested in use in bin.
2021-07-13 13:39:01.532792: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (2048): Total Chunks: 8, Chunks in use: 8. 28.0KiB allocated for chunks. 28.0KiB in use in bin. 27.0KiB client-requested in use in bin.
2021-07-13 13:39:01.532819: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.532846: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.532874: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (16384): Total Chunks: 1, Chunks in use: 0. 18.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.532903: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (32768): Total Chunks: 3, Chunks in use: 3. 108.0KiB allocated for chunks. 108.0KiB in use in bin. 108.0KiB client-requested in use in bin.
2021-07-13 13:39:01.532932: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (65536): Total Chunks: 15, Chunks in use: 15. 1.39MiB allocated for chunks. 1.39MiB in use in bin. 1.31MiB client-requested in use in bin.
2021-07-13 13:39:01.532962: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (131072): Total Chunks: 19, Chunks in use: 18. 3.24MiB allocated for chunks. 3.05MiB in use in bin. 2.87MiB client-requested in use in bin.
2021-07-13 13:39:01.532990: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (262144): Total Chunks: 4, Chunks in use: 4. 1.08MiB allocated for chunks. 1.08MiB in use in bin. 1.08MiB client-requested in use in bin.
2021-07-13 13:39:01.533018: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.533046: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (1048576): Total Chunks: 6, Chunks in use: 4. 8.09MiB allocated for chunks. 5.06MiB in use in bin. 5.06MiB client-requested in use in bin.
2021-07-13 13:39:01.533090: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.534200: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.535103: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.535821: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.536315: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.537217: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-13 13:39:01.538080: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (134217728): Total Chunks: 1, Chunks in use: 1. 139.87MiB allocated for chunks. 139.87MiB in use in bin. 139.87MiB client-requested in use in bin.
2021-07-13 13:39:01.538764: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (268435456): Total Chunks: 2, Chunks in use: 1. 884.13MiB allocated for chunks. 559.46MiB in use in bin. 559.46MiB client-requested in use in bin.
2021-07-13 13:39:01.539305: I tensorflow/core/common_runtime/bfc_allocator.cc:1014] Bin for 13.11GiB was 256.00MiB, Chunk State:
2021-07-13 13:39:01.540161: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Size: 324.67MiB | Requested Size: 0B | in_use: 0 | bin_num: 20, prev: Size: 139.87MiB | Requested Size: 139.87MiB | in_use: 1 | bin_num: -1, for: SameWorkerRecvDone, stepid: 0, last_action: 9223372089016152803, for: UNUSED, stepid: 0, last_action: 0
2021-07-13 13:39:01.540974: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 2097152
2021-07-13 13:39:01.541480: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800000 of size 256 by op CNN1/truncated_normal/stddev action_count 9223372089016152593 step 0 next 1
2021-07-13 13:39:01.542009: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800100 of size 512 by op CNN1/Const action_count 9223372089016152594 step 0 next 2
2021-07-13 13:39:01.542472: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800300 of size 512 by op CNN1/prelu/Const action_count 9223372089016152595 step 0 next 3
2021-07-13 13:39:01.542906: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800500 of size 512 by op CNN2/Const action_count 9223372089016152596 step 0 next 4
2021-07-13 13:39:01.543433: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800700 of size 512 by op CNN2/prelu/Const action_count 9223372089016152597 step 0 next 5
2021-07-13 13:39:01.544111: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800900 of size 256 by op CNN3/truncated_normal/stddev action_count 9223372089016152598 step 0 next 6
2021-07-13 13:39:01.544834: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800a00 of size 512 by op CNN3/Const action_count 9223372089016152599 step 0 next 7
2021-07-13 13:39:01.545294: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800c00 of size 512 by op CNN3/prelu/Const action_count 9223372089016152600 step 0 next 8
2021-07-13 13:39:01.545972: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800e00 of size 256 by op CNN4/truncated_normal/stddev action_count 9223372089016152601 step 0 next 9
2021-07-13 13:39:01.546480: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800f00 of size 512 by op CNN4/Const action_count 9223372089016152602 step 0 next 10
2021-07-13 13:39:01.547094: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801100 of size 512 by op CNN4/prelu/Const action_count 9223372089016152603 step 0 next 11
2021-07-13 13:39:01.547765: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801300 of size 256 by op CNN5/truncated_normal/stddev action_count 9223372089016152604 step 0 next 12
2021-07-13 13:39:01.552928: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801400 of size 256 by op CNN5/Const action_count 9223372089016152605 step 0 next 13
2021-07-13 13:39:01.553615: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801500 of size 256 by op CNN5/prelu/Const action_count 9223372089016152606 step 0 next 14
2021-07-13 13:39:01.554044: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801600 of size 256 by op CNN6/truncated_normal/mean action_count 9223372089016152607 step 0 next 15
2021-07-13 13:39:01.554491: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801700 of size 256 by op CNN6/truncated_normal/stddev action_count 9223372089016152608 step 0 next 16
2021-07-13 13:39:01.554922: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801800 of size 256 by op CNN6/Const action_count 9223372089016152609 step 0 next 17
2021-07-13 13:39:01.555442: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801900 of size 256 by op CNN6/prelu/Const action_count 9223372089016152610 step 0 next 18
2021-07-13 13:39:01.556135: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801a00 of size 256 by op CNN7/truncated_normal/stddev action_count 9223372089016152611 step 0 next 19
2021-07-13 13:39:01.556943: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801b00 of size 256 by op CNN7/Const action_count 9223372089016152612 step 0 next 20
2021-07-13 13:39:01.557439: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801c00 of size 256 by op CNN7/prelu/Const action_count 9223372089016152613 step 0 next 21
2021-07-13 13:39:01.557995: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801d00 of size 256 by op CNN8/truncated_normal/stddev action_count 9223372089016152614 step 0 next 22
2021-07-13 13:39:01.558436: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801e00 of size 256 by op CNN8/Const action_count 9223372089016152615 step 0 next 23
2021-07-13 13:39:01.558867: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801f00 of size 256 by op CNN8/prelu/Const action_count 9223372089016152616 step 0 next 24
2021-07-13 13:39:01.559397: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802000 of size 256 by op A1/Const action_count 9223372089016152617 step 0 next 25
2021-07-13 13:39:01.560092: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802100 of size 256 by op A1/prelu/Const action_count 9223372089016152618 step 0 next 26
2021-07-13 13:39:01.560735: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802200 of size 256 by op B1/truncated_normal/stddev action_count 9223372089016152619 step 0 next 27
2021-07-13 13:39:01.561233: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802300 of size 256 by op B2/truncated_normal/stddev action_count 9223372089016152620 step 0 next 28
2021-07-13 13:39:01.562062: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802400 of size 256 by op B2/Const action_count 9223372089016152621 step 0 next 29
2021-07-13 13:39:01.563144: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802500 of size 256 by op B2/prelu/Const action_count 9223372089016152622 step 0 next 30
2021-07-13 13:39:01.563624: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802600 of size 1536 by op Up-PS/Up-PS_CNN/Const action_count 9223372089016152623 step 0 next 31
2021-07-13 13:39:01.564136: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802c00 of size 256 by op R-CNN1/truncated_normal/stddev action_count 9223372089016152624 step 0 next 32
2021-07-13 13:39:01.567564: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802d00 of size 256 by op beta1_power/Initializer/initial_value action_count 9223372089016152625 step 0 next 33
2021-07-13 13:39:01.567999: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802e00 of size 256 by op beta2_power/Initializer/initial_value action_count 9223372089016152626 step 0 next 34
2021-07-13 13:39:01.568518: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802f00 of size 3584 by op CNN1/conv_W/Adam/Initializer/zeros action_count 9223372089016152627 step 0 next 35
2021-07-13 13:39:01.569056: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d803d00 of size 512 by op CNN1/conv_B/Adam/Initializer/zeros action_count 9223372089016152628 step 0 next 36
2021-07-13 13:39:01.569519: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d803f00 of size 512 by op CNN1/prelu/CNN1_prelu/Adam/Initializer/zeros action_count 9223372089016152629 step 0 next 37
2021-07-13 13:39:01.569953: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804100 of size 512 by op CNN2/conv_B/Adam_1/Initializer/zeros action_count 9223372089016152630 step 0 next 38
2021-07-13 13:39:01.570461: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804300 of size 512 by op CNN2/prelu/CNN2_prelu/Adam/Initializer/zeros action_count 9223372089016152631 step 0 next 39
2021-07-13 13:39:01.570975: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804500 of size 512 by op CNN3/conv_B/Adam/Initializer/zeros action_count 9223372089016152632 step 0 next 40
2021-07-13 13:39:01.571476: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804700 of size 512 by op CNN3/prelu/CNN3_prelu/Adam/Initializer/zeros action_count 9223372089016152633 step 0 next 41
2021-07-13 13:39:01.571964: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804900 of size 512 by op CNN4/conv_B/Adam/Initializer/zeros action_count 9223372089016152634 step 0 next 42
2021-07-13 13:39:01.572442: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804b00 of size 512 by op CNN4/prelu/CNN4_prelu/Adam_1/Initializer/zeros action_count 9223372089016152635 step 0 next 43
2021-07-13 13:39:01.573081: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804d00 of size 256 by op CNN5/conv_B/Adam/Initializer/zeros action_count 9223372089016152636 step 0 next 44
2021-07-13 13:39:01.573541: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804e00 of size 256 by op CNN5/prelu/CNN5_prelu/Adam_1/Initializer/zeros action_count 9223372089016152637 step 0 next 45
2021-07-13 13:39:01.573973: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804f00 of size 256 by op CNN6/conv_B/Adam/Initializer/zeros action_count 9223372089016152638 step 0 next 46
2021-07-13 13:39:01.574493: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805000 of size 256 by op CNN6/prelu/CNN6_prelu/Adam/Initializer/zeros action_count 9223372089016152639 step 0 next 47
2021-07-13 13:39:01.574964: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805100 of size 256 by op CNN7/conv_B/Adam_1/Initializer/zeros action_count 9223372089016152640 step 0 next 48
2021-07-13 13:39:01.575429: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805200 of size 256 by op CNN7/prelu/CNN7_prelu/Adam/Initializer/zeros action_count 9223372089016152641 step 0 next 49
2021-07-13 13:39:01.575860: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805300 of size 256 by op CNN8/conv_B/Adam/Initializer/zeros action_count 9223372089016152642 step 0 next 50
2021-07-13 13:39:01.576297: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805400 of size 256 by op CNN8/prelu/CNN8_prelu/Adam/Initializer/zeros action_count 9223372089016152643 step 0 next 51
2021-07-13 13:39:01.576728: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805500 of size 256 by op A1/conv_B/Adam/Initializer/zeros action_count 9223372089016152644 step 0 next 52
2021-07-13 13:39:01.577175: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805600 of size 256 by op A1/prelu/A1_prelu/Adam_1/Initializer/zeros action_count 9223372089016152645 step 0 next 53
2021-07-13 13:39:01.578059: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805700 of size 256 by op B1/conv_B/Adam/Initializer/zeros action_count 9223372089016152646 step 0 next 54
2021-07-13 13:39:01.578610: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805800 of size 256 by op B1/prelu/B1_prelu/Adam_1/Initializer/zeros action_count 9223372089016152647 step 0 next 55
2021-07-13 13:39:01.579059: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805900 of size 256 by op B2/conv_B/Adam_1/Initializer/zeros action_count 9223372089016152648 step 0 next 56
2021-07-13 13:39:01.579564: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805a00 of size 256 by op B2/prelu/B2_prelu/Adam/Initializer/zeros action_count 9223372089016152649 step 0 next 57
2021-07-13 13:39:01.584305: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805b00 of size 1536 by op Up-PS/Up-PS_CNN/conv_B/Adam/Initializer/zeros action_count 9223372089016152650 step 0 next 58
2021-07-13 13:39:01.585137: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d806100 of size 3584 by op R-CNN1/conv_W/Adam/Initializer/zeros action_count 9223372089016152651 step 0 next 59
2021-07-13 13:39:01.585930: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d806f00 of size 138240 by op A1/conv_W/Adam/Initializer/zeros/_0cf0 action_count 9223372089016152652 step 0 next 60
2021-07-13 13:39:01.586437: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d828b00 of size 69120 by op B1/conv_W/Adam/Initializer/zeros/_2cf2 action_count 9223372089016152653 step 0 next 61
2021-07-13 13:39:01.587031: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d839900 of size 36864 by op B2/conv_W/Adam/Initializer/zeros/_4cf4 action_count 9223372089016152654 step 0 next 62
2021-07-13 13:39:01.587540: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d842900 of size 283392 by op CNN2/conv_W/Adam/Initializer/zeros/_6cf6 action_count 9223372089016152655 step 0 next 63
2021-07-13 13:39:01.587976: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d887c00 of size 221440 by op CNN3/conv_W/Adam/Initializer/zeros/_8cf8 action_count 9223372089016152656 step 0 next 64
2021-07-13 13:39:01.588468: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d8bdd00 of size 183808 by op CNN4/conv_W/Adam_1/Initializer/zeros/_11cf11 action_count 9223372089016152657 step 0 next 65
2021-07-13 13:39:01.588948: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d8eab00 of size 151808 by op CNN5/conv_W/Adam/Initializer/zeros/_12cf12 action_count 9223372089016152658 step 0 next 66
2021-07-13 13:39:01.589386: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d90fc00 of size 127232 by op CNN6/conv_W/Adam/Initializer/zeros/_14cf14 action_count 9223372089016152659 step 0 next 67
2021-07-13 13:39:01.589835: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d92ed00 of size 106752 by op CNN7/conv_W/Adam/Initializer/zeros/_16cf16 action_count 9223372089016152660 step 0 next 68
2021-07-13 13:39:01.590385: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d948e00 of size 89856 by op CNN8/conv_W/Adam/Initializer/zeros/_18cf18 action_count 9223372089016152661 step 0 next 69
2021-07-13 13:39:01.591092: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d95ed00 of size 1280 by op ScratchBuffer action_count 9223372089016152663 step 0 next 72
2021-07-13 13:39:01.591750: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d95f200 of size 127232 by op SameWorkerRecvDone action_count 9223372089016152769 step 0 next 73
2021-07-13 13:39:01.592228: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b0d97e300 of size 196608 by op UNUSED action_count 9223372089016152797 step 1 next 76
2021-07-13 13:39:01.593096: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152738 step 0 next 142
2021-07-13 13:39:01.593628: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152739 step 0 next 143
2021-07-13 13:39:01.594429: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae500 of size 512 by op SameWorkerRecvDone action_count 9223372089016152741 step 0 next 145
2021-07-13 13:39:01.595552: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae700 of size 512 by op SameWorkerRecvDone action_count 9223372089016152742 step 0 next 146
2021-07-13 13:39:01.599105: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae900 of size 512 by op SameWorkerRecvDone action_count 9223372089016152743 step 0 next 147
2021-07-13 13:39:01.599563: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aeb00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152745 step 0 next 149
2021-07-13 13:39:01.600028: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aec00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152747 step 0 next 150
2021-07-13 13:39:01.600464: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aed00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152748 step 0 next 151
2021-07-13 13:39:01.600895: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aee00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152749 step 0 next 152
2021-07-13 13:39:01.601416: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aef00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152750 step 0 next 153
2021-07-13 13:39:01.602057: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af000 of size 512 by op SameWorkerRecvDone action_count 9223372089016152774 step 0 next 168
2021-07-13 13:39:01.602575: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af200 of size 256 by op SameWorkerRecvDone action_count 9223372089016152775 step 0 next 169
2021-07-13 13:39:01.603007: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152776 step 0 next 170
2021-07-13 13:39:01.603447: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152777 step 0 next 171
2021-07-13 13:39:01.603897: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152780 step 0 next 173
2021-07-13 13:39:01.604321: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af600 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152781 step 0 next 174
2021-07-13 13:39:01.604798: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b0400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152783 step 0 next 176
2021-07-13 13:39:01.605245: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b0500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152784 step 0 next 177
2021-07-13 13:39:01.605688: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b0600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152785 step 0 next 178
2021-07-13 13:39:01.606134: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b0700 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152789 step 0 next 180
2021-07-13 13:39:01.606590: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152790 step 0 next 181
2021-07-13 13:39:01.607020: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152791 step 0 next 182
2021-07-13 13:39:01.607524: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1700 of size 256 by op SameWorkerRecvDone action_count 9223372089016152792 step 0 next 183
2021-07-13 13:39:01.607956: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1800 of size 256 by op SameWorkerRecvDone action_count 9223372089016152793 step 0 next 184
2021-07-13 13:39:01.608393: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1900 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152795 step 0 next 186
2021-07-13 13:39:01.608825: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b2700 of size 256 by op CNN1/prelu/mul_1/y action_count 9223372089016152800 step 0 next 74
2021-07-13 13:39:01.609256: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b2800 of size 256 by op CNN4/dropout/Const action_count 9223372089016152801 step 0 next 75
2021-07-13 13:39:01.609689: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b2900 of size 256 by op SameWorkerRecvDone action_count 9223372089016152804 step 0 next 190
2021-07-13 13:39:01.610119: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b2a00 of size 256 by op CNN1/dropout/Sub action_count 9223372089016152805 step 3 next 191
2021-07-13 13:39:01.610560: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b0d9b2b00 of size 18432 by op UNUSED action_count 0 step 0 next 77
2021-07-13 13:39:01.611027: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b7300 of size 69120 by op SameWorkerRecvDone action_count 9223372089016152696 step 0 next 78
2021-07-13 13:39:01.611664: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9c8100 of size 229120 by op SameWorkerRecvDone action_count 9223372089016152778 step 0 next 18446744073709551615
2021-07-13 13:39:01.615903: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 4194304
2021-07-13 13:39:01.616455: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0da00000 of size 1327104 by op Up-PS/Up-PS_CNN/conv_W/Adam/Initializer/zeros/_20cf20 action_count 9223372089016152662 step 0 next 71
2021-07-13 13:39:01.616903: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b0db44000 of size 1330688 by op UNUSED action_count 9223372089016152796 step 1 next 79
2021-07-13 13:39:01.617321: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc88e00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152710 step 0 next 115
2021-07-13 13:39:01.617782: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89000 of size 512 by op SameWorkerRecvDone action_count 9223372089016152711 step 0 next 116
2021-07-13 13:39:01.618224: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89200 of size 512 by op SameWorkerRecvDone action_count 9223372089016152712 step 0 next 117
2021-07-13 13:39:01.618656: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89400 of size 512 by op SameWorkerRecvDone action_count 9223372089016152713 step 0 next 118
2021-07-13 13:39:01.619091: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89600 of size 512 by op SameWorkerRecvDone action_count 9223372089016152714 step 0 next 119
2021-07-13 13:39:01.619542: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89800 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152715 step 0 next 120
2021-07-13 13:39:01.619976: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8a600 of size 512 by op SameWorkerRecvDone action_count 9223372089016152716 step 0 next 121
2021-07-13 13:39:01.620408: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8a800 of size 512 by op SameWorkerRecvDone action_count 9223372089016152717 step 0 next 122
2021-07-13 13:39:01.620839: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8aa00 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152719 step 0 next 124
2021-07-13 13:39:01.621273: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8b800 of size 512 by op SameWorkerRecvDone action_count 9223372089016152720 step 0 next 125
2021-07-13 13:39:01.621706: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8ba00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152721 step 0 next 126
2021-07-13 13:39:01.622143: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8bc00 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152724 step 0 next 129
2021-07-13 13:39:01.622575: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8ca00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152725 step 0 next 130
2021-07-13 13:39:01.623012: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8cc00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152726 step 0 next 131
2021-07-13 13:39:01.623446: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8ce00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152727 step 0 next 132
2021-07-13 13:39:01.623874: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8d000 of size 1536 by op SameWorkerRecvDone action_count 9223372089016152729 step 0 next 134
2021-07-13 13:39:01.624304: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8d600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152730 step 0 next 135
2021-07-13 13:39:01.624736: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8d700 of size 512 by op SameWorkerRecvDone action_count 9223372089016152731 step 0 next 136
2021-07-13 13:39:01.625171: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8d900 of size 512 by op SameWorkerRecvDone action_count 9223372089016152733 step 0 next 138
2021-07-13 13:39:01.625610: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8db00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152734 step 0 next 139
2021-07-13 13:39:01.626049: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8dd00 of size 118016 by op SameWorkerRecvDone action_count 9223372089016152746 step 0 next 80
2021-07-13 13:39:01.626492: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dcaaa00 of size 183808 by op SameWorkerRecvDone action_count 9223372089016152757 step 0 next 81
2021-07-13 13:39:01.626935: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dcd7800 of size 127232 by op SameWorkerRecvDone action_count 9223372089016152759 step 0 next 158
2021-07-13 13:39:01.627373: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dcf6900 of size 156160 by op SameWorkerRecvDone action_count 9223372089016152770 step 0 next 82
2021-07-13 13:39:01.627862: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1cb00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152755 step 0 next 156
2021-07-13 13:39:01.632418: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1cd00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152756 step 0 next 157
2021-07-13 13:39:01.633087: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1cf00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152760 step 0 next 159
2021-07-13 13:39:01.633512: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d000 of size 256 by op SameWorkerRecvDone action_count 9223372089016152761 step 0 next 160
2021-07-13 13:39:01.633974: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d100 of size 256 by op SameWorkerRecvDone action_count 9223372089016152762 step 0 next 161
2021-07-13 13:39:01.634421: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d200 of size 512 by op SameWorkerRecvDone action_count 9223372089016152764 step 0 next 163
2021-07-13 13:39:01.634872: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d400 of size 512 by op SameWorkerRecvDone action_count 9223372089016152765 step 0 next 164
2021-07-13 13:39:01.635307: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152766 step 0 next 165
2021-07-13 13:39:01.635763: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d700 of size 256 by op SameWorkerRecvDone action_count 9223372089016152771 step 0 next 166
2021-07-13 13:39:01.636243: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d800 of size 256 by op SameWorkerRecvDone action_count 9223372089016152772 step 0 next 83
2021-07-13 13:39:01.636675: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d900 of size 106752 by op SameWorkerRecvDone action_count 9223372089016152787 step 0 next 179
2021-07-13 13:39:01.637114: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd37a00 of size 114688 by op SameWorkerRecvDone action_count 9223372089016152788 step 0 next 84
2021-07-13 13:39:01.637544: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd53a00 of size 1536 by op SameWorkerRecvDone action_count 9223372089016152677 step 0 next 85
2021-07-13 13:39:01.637977: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd54000 of size 256 by op SameWorkerRecvDone action_count 9223372089016152678 step 0 next 86
2021-07-13 13:39:01.638407: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd54100 of size 256 by op SameWorkerRecvDone action_count 9223372089016152679 step 0 next 87
2021-07-13 13:39:01.638840: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd54200 of size 138240 by op SameWorkerRecvDone action_count 9223372089016152680 step 0 next 88
2021-07-13 13:39:01.639269: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd75e00 of size 1536 by op SameWorkerRecvDone action_count 9223372089016152681 step 0 next 89
2021-07-13 13:39:01.639700: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd76400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152682 step 0 next 90
2021-07-13 13:39:01.640130: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd76500 of size 138240 by op SameWorkerRecvDone action_count 9223372089016152683 step 0 next 91
2021-07-13 13:39:01.640561: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98100 of size 256 by op SameWorkerRecvDone action_count 9223372089016152684 step 0 next 92
2021-07-13 13:39:01.640990: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98200 of size 256 by op SameWorkerRecvDone action_count 9223372089016152685 step 0 next 93
2021-07-13 13:39:01.641422: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152686 step 0 next 94
2021-07-13 13:39:01.641851: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152687 step 0 next 95
2021-07-13 13:39:01.642286: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152688 step 0 next 96
2021-07-13 13:39:01.642715: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152689 step 0 next 97
2021-07-13 13:39:01.643162: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98700 of size 138240 by op SameWorkerRecvDone action_count 9223372089016152690 step 0 next 98
2021-07-13 13:39:01.646865: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddba300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152691 step 0 next 99
2021-07-13 13:39:01.647418: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddba400 of size 69120 by op SameWorkerRecvDone action_count 9223372089016152692 step 0 next 100
2021-07-13 13:39:01.647853: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb200 of size 256 by op SameWorkerRecvDone action_count 9223372089016152693 step 0 next 101
2021-07-13 13:39:01.648308: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152694 step 0 next 102
2021-07-13 13:39:01.648757: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152697 step 0 next 103
2021-07-13 13:39:01.649212: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152698 step 0 next 104
2021-07-13 13:39:01.649645: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb600 of size 69120 by op SameWorkerRecvDone action_count 9223372089016152699 step 0 next 105
2021-07-13 13:39:01.650080: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dddc400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152700 step 0 next 106
2021-07-13 13:39:01.650512: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dddc500 of size 36864 by op SameWorkerRecvDone action_count 9223372089016152701 step 0 next 107
2021-07-13 13:39:01.650943: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dde5500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152702 step 0 next 108
2021-07-13 13:39:01.651374: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dde5600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152703 step 0 next 109
2021-07-13 13:39:01.651802: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dde5700 of size 36864 by op SameWorkerRecvDone action_count 9223372089016152704 step 0 next 110
2021-07-13 13:39:01.652233: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddee700 of size 256 by op SameWorkerRecvDone action_count 9223372089016152705 step 0 next 111
2021-07-13 13:39:01.652663: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddee800 of size 71680 by op SameWorkerRecvDone action_count 9223372089016152706 step 0 next 18446744073709551615
2021-07-13 13:39:01.653096: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 8388608
2021-07-13 13:39:01.653522: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de00000 of size 256 by op SameWorkerRecvDone action_count 9223372089016152707 step 0 next 113
2021-07-13 13:39:01.653952: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de00100 of size 512 by op SameWorkerRecvDone action_count 9223372089016152708 step 0 next 114
2021-07-13 13:39:01.654383: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de00300 of size 221440 by op SameWorkerRecvDone action_count 9223372089016152718 step 0 next 123
2021-07-13 13:39:01.654814: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de36400 of size 221440 by op SameWorkerRecvDone action_count 9223372089016152722 step 0 next 127
2021-07-13 13:39:01.655244: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de6c500 of size 183808 by op SameWorkerRecvDone action_count 9223372089016152723 step 0 next 128
2021-07-13 13:39:01.655679: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de99300 of size 183808 by op SameWorkerRecvDone action_count 9223372089016152728 step 0 next 133
2021-07-13 13:39:01.656108: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dec6100 of size 283392 by op SameWorkerRecvDone action_count 9223372089016152732 step 0 next 137
2021-07-13 13:39:01.656538: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0df0b400 of size 151808 by op SameWorkerRecvDone action_count 9223372089016152736 step 0 next 140
2021-07-13 13:39:01.656967: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0df30500 of size 183808 by op SameWorkerRecvDone action_count 9223372089016152737 step 0 next 141
2021-07-13 13:39:01.657400: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0df5d300 of size 283392 by op SameWorkerRecvDone action_count 9223372089016152740 step 0 next 144
2021-07-13 13:39:01.657830: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dfa2600 of size 151808 by op SameWorkerRecvDone action_count 9223372089016152744 step 0 next 148
2021-07-13 13:39:01.658265: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dfc7700 of size 283392 by op SameWorkerRecvDone action_count 9223372089016152752 step 0 next 154
2021-07-13 13:39:01.662505: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e00ca00 of size 221440 by op SameWorkerRecvDone action_count 9223372089016152753 step 0 next 155
2021-07-13 13:39:01.663235: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e042b00 of size 1327104 by op SameWorkerRecvDone action_count 9223372089016152763 step 0 next 162
2021-07-13 13:39:01.663670: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e186b00 of size 1327104 by op SameWorkerRecvDone action_count 9223372089016152773 step 0 next 167
2021-07-13 13:39:01.664126: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e2cab00 of size 106752 by op SameWorkerRecvDone action_count 9223372089016152779 step 0 next 172
2021-07-13 13:39:01.664574: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e2e4c00 of size 1327104 by op SameWorkerRecvDone action_count 9223372089016152782 step 0 next 175
2021-07-13 13:39:01.665029: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e428c00 of size 89856 by op SameWorkerRecvDone action_count 9223372089016152794 step 0 next 185
2021-07-13 13:39:01.665466: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b0e43eb00 of size 1840384 by op UNUSED action_count 0 step 0 next 18446744073709551615
2021-07-13 13:39:01.665901: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 1073741824
2021-07-13 13:39:01.666336: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e600000 of size 586641152 by op SameWorkerRecvDone action_count 9223372089016152802 step 0 next 188
2021-07-13 13:39:01.666768: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b31576f00 of size 146660352 by op SameWorkerRecvDone action_count 9223372089016152803 step 0 next 189
2021-07-13 13:39:01.667210: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b3a154b00 of size 340440320 by op UNUSED action_count 0 step 0 next 18446744073709551615
2021-07-13 13:39:01.667639: I tensorflow/core/common_runtime/bfc_allocator.cc:1051] Summary of in-use Chunks by size:
2021-07-13 13:39:01.668080: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 87 Chunks of size 256 totalling 21.8KiB
2021-07-13 13:39:01.668502: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 40 Chunks of size 512 totalling 20.0KiB
2021-07-13 13:39:01.668930: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1280 totalling 1.2KiB
2021-07-13 13:39:01.669357: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 1536 totalling 7.5KiB
2021-07-13 13:39:01.669785: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 8 Chunks of size 3584 totalling 28.0KiB
2021-07-13 13:39:01.670214: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 36864 totalling 108.0KiB
2021-07-13 13:39:01.670641: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 69120 totalling 270.0KiB
2021-07-13 13:39:01.671074: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 71680 totalling 70.0KiB
2021-07-13 13:39:01.671501: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 89856 totalling 175.5KiB
2021-07-13 13:39:01.671929: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 106752 totalling 312.8KiB
2021-07-13 13:39:01.672362: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 114688 totalling 112.0KiB
2021-07-13 13:39:01.672789: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 118016 totalling 115.2KiB
2021-07-13 13:39:01.673219: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 127232 totalling 372.8KiB
2021-07-13 13:39:01.673645: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 138240 totalling 540.0KiB
2021-07-13 13:39:01.674077: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 151808 totalling 444.8KiB
2021-07-13 13:39:01.678182: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 156160 totalling 152.5KiB
2021-07-13 13:39:01.679297: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 183808 totalling 897.5KiB
2021-07-13 13:39:01.679733: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 221440 totalling 865.0KiB
2021-07-13 13:39:01.680204: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 229120 totalling 223.8KiB
2021-07-13 13:39:01.680680: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 283392 totalling 1.08MiB
2021-07-13 13:39:01.681195: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 1327104 totalling 5.06MiB
2021-07-13 13:39:01.681690: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 146660352 totalling 139.87MiB
2021-07-13 13:39:01.682223: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 586641152 totalling 559.46MiB
2021-07-13 13:39:01.682646: I tensorflow/core/common_runtime/bfc_allocator.cc:1058] Sum Total of in-use chunks: 710.10MiB
2021-07-13 13:39:01.683099: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] total_region_allocatedbytes: 1088421888 memorylimit: 6292504576 available bytes: 5204082688 curr_region_allocationbytes: 1073741824
2021-07-13 13:39:01.683561: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Stats:
Limit: 6292504576
InUse: 744595456
MaxInUse: 744595456
NumAllocs: 200
MaxAllocSize: 586641152
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2021-07-13 13:39:01.684082: W tensorflow/core/common_runtime/bfc_allocator.cc:467] *****___
2021-07-13 13:39:01.684564: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_ops.cc:662 : Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call
return fn(*args)
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1359, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1451, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node CNN1/CNN1_conv}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[output/_227]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node CNN1/CNN1_conv}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Code\MAI\Source\Services\DCSCN\sr.py", line 51, in
tf.app.run()
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\absl\app.py", line 312, in run
_run_main(main, args)
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\absl\app.py", line 258, in _run_main
sys.exit(main(argv))
File "C:\Code\MAI\Source\Services\DCSCN\sr.py", line 47, in main
model.do_for_file(FLAGS.file, FLAGS.output_dir)
File "C:\Code\MAI\Source\Services\DCSCN\DCSCN.py", line 618, in do_for_file
output_y_image = self.do(input_yimage)
File "C:\Code\MAI\Source\Services\DCSCN\DCSCN.py", line 580, in do
y = self.sess.run(self.y, feed_dict={self.x: image.reshape(1, image.shape[0], image.shape[1], ch),
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1368, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node CNN1/CNN1_conv (defined at Code\MAI\Source\Services\DCSCN\helper\tf_graph.py:105) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[output/_227]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node CNN1/CNN1_conv (defined at Code\MAI\Source\Services\DCSCN\helper\tf_graph.py:105) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Errors may have originated from an input operation.
Input Source operations connected to node CNN1/CNN1_conv:
x (defined at Code\MAI\Source\Services\DCSCN\DCSCN.py:227)
Input Source operations connected to node CNN1/CNN1_conv:
x (defined at Code\MAI\Source\Services\DCSCN\DCSCN.py:227)
Original stack trace for 'CNN1/CNN1_conv':
File "Code\MAI\Source\Services\DCSCN\sr.py", line 51, in
tf.app.run()
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\absl\app.py", line 312, in run
_run_main(main, args)
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\absl\app.py", line 258, in _run_main
sys.exit(main(argv))
File "Code\MAI\Source\Services\DCSCN\sr.py", line 40, in main
model.build_graph()
File "Code\MAI\Source\Services\DCSCN\DCSCN.py", line 255, in build_graph
self.build_conv("CNN%d" % (i + 1), input_tensor, self.cnn_size, input_feature_num,
File "Code\MAI\Source\Services\DCSCN\helper\tf_graph.py", line 124, in build_conv
h = self.conv2d(input_tensor, w, self.cnn_stride, bias=b, use_batch_norm=use_batch_norm, name=name)
File "Code\MAI\Source\Services\DCSCN\helper\tf_graph.py", line 105, in conv2d
output = tf.nn.conv2d(input_tensor, w, strides=[1, stride, stride, 1], padding="SAME", name=name + "_conv")
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper
return target(*args, **kwargs)
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 2269, in conv2d
return gen_nn_ops.conv2d(
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\ops\gen_nnops.py", line 968, in conv2d
, _, _op, _outputs = _op_def_library._apply_op_helper(
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 748, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\framework\ops.py", line 3557, in _create_op_internal
ret = Operation(
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\framework\ops.py", line 2045, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)
I've tried 10x, 5x, 2x and i'm receiving the same error. the image is 7995x4586 It also doesn't seem to be using the GPU, I'm not sure if it's designed to or not with the sr.py These are the call i am using python train.py --dataset=hr384 --layers=8 --filters=96 --training_images=100 --scale 2 python sr.py --file=SatImageSm.png --layers=8 --filters=96 --training_images=100 --scale 2
DCSCN v2------------------------------------- 2021/7/13 13:35:48 [dcscn_L8_F96to48_NIN_A64_PS_R1F32] Feature:96 82 75 68 62 57 52 48 Total: (540)64 32 32 384 1 Complexity:687,268 Receptive Fields:21 CNN1/conv_W:0 (3, 3, 1, 96)=864, CNN1/conv_B:0 96, CNN1/prelu/CNN1_prelu:0 96, CNN2/conv_W:0 (3, 3, 96, 82)=70848, CNN2/conv_B:0 82, CNN2/prelu/CNN2_prelu:0 82, CNN3/conv_W:0 (3, 3, 82, 75)=55350, CNN3/conv_B:0 75, CNN3/prelu/CNN3_prelu:0 75, CNN4/conv_W:0 (3, 3, 75, 68)=45900, CNN4/conv_B:0 68, CNN4/prelu/CNN4_prelu:0 68, CNN5/conv_W:0 (3, 3, 68, 62)=37944, CNN5/conv_B:0 62, CNN5/prelu/CNN5_prelu:0 62, CNN6/conv_W:0 (3, 3, 62, 57)=31806, CNN6/conv_B:0 57, CNN6/prelu/CNN6_prelu:0 57, CNN7/conv_W:0 (3, 3, 57, 52)=26676, CNN7/conv_B:0 52, CNN7/prelu/CNN7_prelu:0 52, CNN8/conv_W:0 (3, 3, 52, 48)=22464, CNN8/conv_B:0 48, CNN8/prelu/CNN8_prelu:0 48, A1/conv_W:0 (1, 1, 540, 64)=34560, A1/conv_B:0 64, A1/prelu/A1_prelu:0 64, B1/conv_W:0 (1, 1, 540, 32)=17280, B1/conv_B:0 32, B1/prelu/B1_prelu:0 32, B2/conv_W:0 (3, 3, 32, 32)=9216, B2/conv_B:0 32, B2/prelu/B2_prelu:0 32, Up-PS/Up-PS_CNN/conv_W:0 (3, 3, 96, 384)=331776, Up-PS/Up-PS_CNN/conv_B:0 384, R-CNN1/conv_W:0 (3, 3, 96, 1)=864, Total 36 variables, 687,268 params Model initialized. Model restored [ models/dcscn_L8_F96to48_NIN_A64_PS_R1F32.ckpt ]. Loaded [SatImageSm.png]: 7995 x 4586 x 3 Saved [output/dcscn_L8_F96to48_NIN_A64_PS_R1F32/SatImageSm.png] Saved [output/dcscn_L8_F96to48_NIN_A64_PS_R1F32/SatImageSm_bicubic.png] Saved [output/dcscn_L8_F96to48_NIN_A64_PS_R1F32/SatImageSm_bicubic_y.png] 2021-07-13 13:39:01.532171: W tensorflow/core/common_runtime/bfc_allocator.cc:456] Allocator (GPU_0_bfc) ran out of memory trying to allocate 13.11GiB (rounded to 14079386880)requested by op CNN1/CNN1_conv If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. Current allocation summary follows. 2021-07-13 13:39:01.532246: I tensorflow/core/common_runtime/bfc_allocator.cc:991] BFCAllocator dump for GPU_0_bfc 2021-07-13 13:39:01.532704: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (256): Total Chunks: 87, Chunks in use: 87. 21.8KiB allocated for chunks. 21.8KiB in use in bin. 13.4KiB client-requested in use in bin. 2021-07-13 13:39:01.532734: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 12.5KiB client-requested in use in bin. 2021-07-13 13:39:01.532763: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (1024): Total Chunks: 6, Chunks in use: 6. 8.8KiB allocated for chunks. 8.8KiB in use in bin. 8.5KiB client-requested in use in bin. 2021-07-13 13:39:01.532792: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (2048): Total Chunks: 8, Chunks in use: 8. 28.0KiB allocated for chunks. 28.0KiB in use in bin. 27.0KiB client-requested in use in bin. 2021-07-13 13:39:01.532819: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.532846: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.532874: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (16384): Total Chunks: 1, Chunks in use: 0. 18.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.532903: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (32768): Total Chunks: 3, Chunks in use: 3. 108.0KiB allocated for chunks. 108.0KiB in use in bin. 108.0KiB client-requested in use in bin. 2021-07-13 13:39:01.532932: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (65536): Total Chunks: 15, Chunks in use: 15. 1.39MiB allocated for chunks. 1.39MiB in use in bin. 1.31MiB client-requested in use in bin. 2021-07-13 13:39:01.532962: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (131072): Total Chunks: 19, Chunks in use: 18. 3.24MiB allocated for chunks. 3.05MiB in use in bin. 2.87MiB client-requested in use in bin. 2021-07-13 13:39:01.532990: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (262144): Total Chunks: 4, Chunks in use: 4. 1.08MiB allocated for chunks. 1.08MiB in use in bin. 1.08MiB client-requested in use in bin. 2021-07-13 13:39:01.533018: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.533046: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (1048576): Total Chunks: 6, Chunks in use: 4. 8.09MiB allocated for chunks. 5.06MiB in use in bin. 5.06MiB client-requested in use in bin. 2021-07-13 13:39:01.533090: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.534200: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.535103: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.535821: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.536315: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.537217: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-07-13 13:39:01.538080: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (134217728): Total Chunks: 1, Chunks in use: 1. 139.87MiB allocated for chunks. 139.87MiB in use in bin. 139.87MiB client-requested in use in bin. 2021-07-13 13:39:01.538764: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (268435456): Total Chunks: 2, Chunks in use: 1. 884.13MiB allocated for chunks. 559.46MiB in use in bin. 559.46MiB client-requested in use in bin. 2021-07-13 13:39:01.539305: I tensorflow/core/common_runtime/bfc_allocator.cc:1014] Bin for 13.11GiB was 256.00MiB, Chunk State: 2021-07-13 13:39:01.540161: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Size: 324.67MiB | Requested Size: 0B | in_use: 0 | bin_num: 20, prev: Size: 139.87MiB | Requested Size: 139.87MiB | in_use: 1 | bin_num: -1, for: SameWorkerRecvDone, stepid: 0, last_action: 9223372089016152803, for: UNUSED, stepid: 0, last_action: 0 2021-07-13 13:39:01.540974: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 2097152 2021-07-13 13:39:01.541480: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800000 of size 256 by op CNN1/truncated_normal/stddev action_count 9223372089016152593 step 0 next 1 2021-07-13 13:39:01.542009: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800100 of size 512 by op CNN1/Const action_count 9223372089016152594 step 0 next 2 2021-07-13 13:39:01.542472: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800300 of size 512 by op CNN1/prelu/Const action_count 9223372089016152595 step 0 next 3 2021-07-13 13:39:01.542906: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800500 of size 512 by op CNN2/Const action_count 9223372089016152596 step 0 next 4 2021-07-13 13:39:01.543433: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800700 of size 512 by op CNN2/prelu/Const action_count 9223372089016152597 step 0 next 5 2021-07-13 13:39:01.544111: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800900 of size 256 by op CNN3/truncated_normal/stddev action_count 9223372089016152598 step 0 next 6 2021-07-13 13:39:01.544834: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800a00 of size 512 by op CNN3/Const action_count 9223372089016152599 step 0 next 7 2021-07-13 13:39:01.545294: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800c00 of size 512 by op CNN3/prelu/Const action_count 9223372089016152600 step 0 next 8 2021-07-13 13:39:01.545972: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800e00 of size 256 by op CNN4/truncated_normal/stddev action_count 9223372089016152601 step 0 next 9 2021-07-13 13:39:01.546480: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d800f00 of size 512 by op CNN4/Const action_count 9223372089016152602 step 0 next 10 2021-07-13 13:39:01.547094: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801100 of size 512 by op CNN4/prelu/Const action_count 9223372089016152603 step 0 next 11 2021-07-13 13:39:01.547765: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801300 of size 256 by op CNN5/truncated_normal/stddev action_count 9223372089016152604 step 0 next 12 2021-07-13 13:39:01.552928: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801400 of size 256 by op CNN5/Const action_count 9223372089016152605 step 0 next 13 2021-07-13 13:39:01.553615: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801500 of size 256 by op CNN5/prelu/Const action_count 9223372089016152606 step 0 next 14 2021-07-13 13:39:01.554044: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801600 of size 256 by op CNN6/truncated_normal/mean action_count 9223372089016152607 step 0 next 15 2021-07-13 13:39:01.554491: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801700 of size 256 by op CNN6/truncated_normal/stddev action_count 9223372089016152608 step 0 next 16 2021-07-13 13:39:01.554922: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801800 of size 256 by op CNN6/Const action_count 9223372089016152609 step 0 next 17 2021-07-13 13:39:01.555442: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801900 of size 256 by op CNN6/prelu/Const action_count 9223372089016152610 step 0 next 18 2021-07-13 13:39:01.556135: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801a00 of size 256 by op CNN7/truncated_normal/stddev action_count 9223372089016152611 step 0 next 19 2021-07-13 13:39:01.556943: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801b00 of size 256 by op CNN7/Const action_count 9223372089016152612 step 0 next 20 2021-07-13 13:39:01.557439: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801c00 of size 256 by op CNN7/prelu/Const action_count 9223372089016152613 step 0 next 21 2021-07-13 13:39:01.557995: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801d00 of size 256 by op CNN8/truncated_normal/stddev action_count 9223372089016152614 step 0 next 22 2021-07-13 13:39:01.558436: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801e00 of size 256 by op CNN8/Const action_count 9223372089016152615 step 0 next 23 2021-07-13 13:39:01.558867: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d801f00 of size 256 by op CNN8/prelu/Const action_count 9223372089016152616 step 0 next 24 2021-07-13 13:39:01.559397: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802000 of size 256 by op A1/Const action_count 9223372089016152617 step 0 next 25 2021-07-13 13:39:01.560092: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802100 of size 256 by op A1/prelu/Const action_count 9223372089016152618 step 0 next 26 2021-07-13 13:39:01.560735: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802200 of size 256 by op B1/truncated_normal/stddev action_count 9223372089016152619 step 0 next 27 2021-07-13 13:39:01.561233: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802300 of size 256 by op B2/truncated_normal/stddev action_count 9223372089016152620 step 0 next 28 2021-07-13 13:39:01.562062: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802400 of size 256 by op B2/Const action_count 9223372089016152621 step 0 next 29 2021-07-13 13:39:01.563144: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802500 of size 256 by op B2/prelu/Const action_count 9223372089016152622 step 0 next 30 2021-07-13 13:39:01.563624: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802600 of size 1536 by op Up-PS/Up-PS_CNN/Const action_count 9223372089016152623 step 0 next 31 2021-07-13 13:39:01.564136: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802c00 of size 256 by op R-CNN1/truncated_normal/stddev action_count 9223372089016152624 step 0 next 32 2021-07-13 13:39:01.567564: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802d00 of size 256 by op beta1_power/Initializer/initial_value action_count 9223372089016152625 step 0 next 33 2021-07-13 13:39:01.567999: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802e00 of size 256 by op beta2_power/Initializer/initial_value action_count 9223372089016152626 step 0 next 34 2021-07-13 13:39:01.568518: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d802f00 of size 3584 by op CNN1/conv_W/Adam/Initializer/zeros action_count 9223372089016152627 step 0 next 35 2021-07-13 13:39:01.569056: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d803d00 of size 512 by op CNN1/conv_B/Adam/Initializer/zeros action_count 9223372089016152628 step 0 next 36 2021-07-13 13:39:01.569519: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d803f00 of size 512 by op CNN1/prelu/CNN1_prelu/Adam/Initializer/zeros action_count 9223372089016152629 step 0 next 37 2021-07-13 13:39:01.569953: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804100 of size 512 by op CNN2/conv_B/Adam_1/Initializer/zeros action_count 9223372089016152630 step 0 next 38 2021-07-13 13:39:01.570461: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804300 of size 512 by op CNN2/prelu/CNN2_prelu/Adam/Initializer/zeros action_count 9223372089016152631 step 0 next 39 2021-07-13 13:39:01.570975: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804500 of size 512 by op CNN3/conv_B/Adam/Initializer/zeros action_count 9223372089016152632 step 0 next 40 2021-07-13 13:39:01.571476: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804700 of size 512 by op CNN3/prelu/CNN3_prelu/Adam/Initializer/zeros action_count 9223372089016152633 step 0 next 41 2021-07-13 13:39:01.571964: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804900 of size 512 by op CNN4/conv_B/Adam/Initializer/zeros action_count 9223372089016152634 step 0 next 42 2021-07-13 13:39:01.572442: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804b00 of size 512 by op CNN4/prelu/CNN4_prelu/Adam_1/Initializer/zeros action_count 9223372089016152635 step 0 next 43 2021-07-13 13:39:01.573081: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804d00 of size 256 by op CNN5/conv_B/Adam/Initializer/zeros action_count 9223372089016152636 step 0 next 44 2021-07-13 13:39:01.573541: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804e00 of size 256 by op CNN5/prelu/CNN5_prelu/Adam_1/Initializer/zeros action_count 9223372089016152637 step 0 next 45 2021-07-13 13:39:01.573973: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d804f00 of size 256 by op CNN6/conv_B/Adam/Initializer/zeros action_count 9223372089016152638 step 0 next 46 2021-07-13 13:39:01.574493: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805000 of size 256 by op CNN6/prelu/CNN6_prelu/Adam/Initializer/zeros action_count 9223372089016152639 step 0 next 47 2021-07-13 13:39:01.574964: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805100 of size 256 by op CNN7/conv_B/Adam_1/Initializer/zeros action_count 9223372089016152640 step 0 next 48 2021-07-13 13:39:01.575429: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805200 of size 256 by op CNN7/prelu/CNN7_prelu/Adam/Initializer/zeros action_count 9223372089016152641 step 0 next 49 2021-07-13 13:39:01.575860: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805300 of size 256 by op CNN8/conv_B/Adam/Initializer/zeros action_count 9223372089016152642 step 0 next 50 2021-07-13 13:39:01.576297: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805400 of size 256 by op CNN8/prelu/CNN8_prelu/Adam/Initializer/zeros action_count 9223372089016152643 step 0 next 51 2021-07-13 13:39:01.576728: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805500 of size 256 by op A1/conv_B/Adam/Initializer/zeros action_count 9223372089016152644 step 0 next 52 2021-07-13 13:39:01.577175: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805600 of size 256 by op A1/prelu/A1_prelu/Adam_1/Initializer/zeros action_count 9223372089016152645 step 0 next 53 2021-07-13 13:39:01.578059: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805700 of size 256 by op B1/conv_B/Adam/Initializer/zeros action_count 9223372089016152646 step 0 next 54 2021-07-13 13:39:01.578610: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805800 of size 256 by op B1/prelu/B1_prelu/Adam_1/Initializer/zeros action_count 9223372089016152647 step 0 next 55 2021-07-13 13:39:01.579059: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805900 of size 256 by op B2/conv_B/Adam_1/Initializer/zeros action_count 9223372089016152648 step 0 next 56 2021-07-13 13:39:01.579564: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805a00 of size 256 by op B2/prelu/B2_prelu/Adam/Initializer/zeros action_count 9223372089016152649 step 0 next 57 2021-07-13 13:39:01.584305: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d805b00 of size 1536 by op Up-PS/Up-PS_CNN/conv_B/Adam/Initializer/zeros action_count 9223372089016152650 step 0 next 58 2021-07-13 13:39:01.585137: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d806100 of size 3584 by op R-CNN1/conv_W/Adam/Initializer/zeros action_count 9223372089016152651 step 0 next 59 2021-07-13 13:39:01.585930: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d806f00 of size 138240 by op A1/conv_W/Adam/Initializer/zeros/_0cf0 action_count 9223372089016152652 step 0 next 60 2021-07-13 13:39:01.586437: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d828b00 of size 69120 by op B1/conv_W/Adam/Initializer/zeros/_2cf2 action_count 9223372089016152653 step 0 next 61 2021-07-13 13:39:01.587031: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d839900 of size 36864 by op B2/conv_W/Adam/Initializer/zeros/_4cf4 action_count 9223372089016152654 step 0 next 62 2021-07-13 13:39:01.587540: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d842900 of size 283392 by op CNN2/conv_W/Adam/Initializer/zeros/_6cf6 action_count 9223372089016152655 step 0 next 63 2021-07-13 13:39:01.587976: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d887c00 of size 221440 by op CNN3/conv_W/Adam/Initializer/zeros/_8cf8 action_count 9223372089016152656 step 0 next 64 2021-07-13 13:39:01.588468: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d8bdd00 of size 183808 by op CNN4/conv_W/Adam_1/Initializer/zeros/_11cf11 action_count 9223372089016152657 step 0 next 65 2021-07-13 13:39:01.588948: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d8eab00 of size 151808 by op CNN5/conv_W/Adam/Initializer/zeros/_12cf12 action_count 9223372089016152658 step 0 next 66 2021-07-13 13:39:01.589386: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d90fc00 of size 127232 by op CNN6/conv_W/Adam/Initializer/zeros/_14cf14 action_count 9223372089016152659 step 0 next 67 2021-07-13 13:39:01.589835: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d92ed00 of size 106752 by op CNN7/conv_W/Adam/Initializer/zeros/_16cf16 action_count 9223372089016152660 step 0 next 68 2021-07-13 13:39:01.590385: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d948e00 of size 89856 by op CNN8/conv_W/Adam/Initializer/zeros/_18cf18 action_count 9223372089016152661 step 0 next 69 2021-07-13 13:39:01.591092: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d95ed00 of size 1280 by op ScratchBuffer action_count 9223372089016152663 step 0 next 72 2021-07-13 13:39:01.591750: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d95f200 of size 127232 by op SameWorkerRecvDone action_count 9223372089016152769 step 0 next 73 2021-07-13 13:39:01.592228: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b0d97e300 of size 196608 by op UNUSED action_count 9223372089016152797 step 1 next 76 2021-07-13 13:39:01.593096: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152738 step 0 next 142 2021-07-13 13:39:01.593628: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152739 step 0 next 143 2021-07-13 13:39:01.594429: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae500 of size 512 by op SameWorkerRecvDone action_count 9223372089016152741 step 0 next 145 2021-07-13 13:39:01.595552: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae700 of size 512 by op SameWorkerRecvDone action_count 9223372089016152742 step 0 next 146 2021-07-13 13:39:01.599105: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9ae900 of size 512 by op SameWorkerRecvDone action_count 9223372089016152743 step 0 next 147 2021-07-13 13:39:01.599563: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aeb00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152745 step 0 next 149 2021-07-13 13:39:01.600028: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aec00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152747 step 0 next 150 2021-07-13 13:39:01.600464: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aed00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152748 step 0 next 151 2021-07-13 13:39:01.600895: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aee00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152749 step 0 next 152 2021-07-13 13:39:01.601416: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9aef00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152750 step 0 next 153 2021-07-13 13:39:01.602057: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af000 of size 512 by op SameWorkerRecvDone action_count 9223372089016152774 step 0 next 168 2021-07-13 13:39:01.602575: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af200 of size 256 by op SameWorkerRecvDone action_count 9223372089016152775 step 0 next 169 2021-07-13 13:39:01.603007: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152776 step 0 next 170 2021-07-13 13:39:01.603447: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152777 step 0 next 171 2021-07-13 13:39:01.603897: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152780 step 0 next 173 2021-07-13 13:39:01.604321: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9af600 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152781 step 0 next 174 2021-07-13 13:39:01.604798: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b0400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152783 step 0 next 176 2021-07-13 13:39:01.605245: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b0500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152784 step 0 next 177 2021-07-13 13:39:01.605688: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b0600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152785 step 0 next 178 2021-07-13 13:39:01.606134: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b0700 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152789 step 0 next 180 2021-07-13 13:39:01.606590: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152790 step 0 next 181 2021-07-13 13:39:01.607020: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152791 step 0 next 182 2021-07-13 13:39:01.607524: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1700 of size 256 by op SameWorkerRecvDone action_count 9223372089016152792 step 0 next 183 2021-07-13 13:39:01.607956: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1800 of size 256 by op SameWorkerRecvDone action_count 9223372089016152793 step 0 next 184 2021-07-13 13:39:01.608393: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b1900 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152795 step 0 next 186 2021-07-13 13:39:01.608825: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b2700 of size 256 by op CNN1/prelu/mul_1/y action_count 9223372089016152800 step 0 next 74 2021-07-13 13:39:01.609256: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b2800 of size 256 by op CNN4/dropout/Const action_count 9223372089016152801 step 0 next 75 2021-07-13 13:39:01.609689: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b2900 of size 256 by op SameWorkerRecvDone action_count 9223372089016152804 step 0 next 190 2021-07-13 13:39:01.610119: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b2a00 of size 256 by op CNN1/dropout/Sub action_count 9223372089016152805 step 3 next 191 2021-07-13 13:39:01.610560: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b0d9b2b00 of size 18432 by op UNUSED action_count 0 step 0 next 77 2021-07-13 13:39:01.611027: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9b7300 of size 69120 by op SameWorkerRecvDone action_count 9223372089016152696 step 0 next 78 2021-07-13 13:39:01.611664: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0d9c8100 of size 229120 by op SameWorkerRecvDone action_count 9223372089016152778 step 0 next 18446744073709551615 2021-07-13 13:39:01.615903: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 4194304 2021-07-13 13:39:01.616455: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0da00000 of size 1327104 by op Up-PS/Up-PS_CNN/conv_W/Adam/Initializer/zeros/_20cf20 action_count 9223372089016152662 step 0 next 71 2021-07-13 13:39:01.616903: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b0db44000 of size 1330688 by op UNUSED action_count 9223372089016152796 step 1 next 79 2021-07-13 13:39:01.617321: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc88e00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152710 step 0 next 115 2021-07-13 13:39:01.617782: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89000 of size 512 by op SameWorkerRecvDone action_count 9223372089016152711 step 0 next 116 2021-07-13 13:39:01.618224: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89200 of size 512 by op SameWorkerRecvDone action_count 9223372089016152712 step 0 next 117 2021-07-13 13:39:01.618656: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89400 of size 512 by op SameWorkerRecvDone action_count 9223372089016152713 step 0 next 118 2021-07-13 13:39:01.619091: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89600 of size 512 by op SameWorkerRecvDone action_count 9223372089016152714 step 0 next 119 2021-07-13 13:39:01.619542: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc89800 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152715 step 0 next 120 2021-07-13 13:39:01.619976: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8a600 of size 512 by op SameWorkerRecvDone action_count 9223372089016152716 step 0 next 121 2021-07-13 13:39:01.620408: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8a800 of size 512 by op SameWorkerRecvDone action_count 9223372089016152717 step 0 next 122 2021-07-13 13:39:01.620839: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8aa00 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152719 step 0 next 124 2021-07-13 13:39:01.621273: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8b800 of size 512 by op SameWorkerRecvDone action_count 9223372089016152720 step 0 next 125 2021-07-13 13:39:01.621706: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8ba00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152721 step 0 next 126 2021-07-13 13:39:01.622143: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8bc00 of size 3584 by op SameWorkerRecvDone action_count 9223372089016152724 step 0 next 129 2021-07-13 13:39:01.622575: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8ca00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152725 step 0 next 130 2021-07-13 13:39:01.623012: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8cc00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152726 step 0 next 131 2021-07-13 13:39:01.623446: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8ce00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152727 step 0 next 132 2021-07-13 13:39:01.623874: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8d000 of size 1536 by op SameWorkerRecvDone action_count 9223372089016152729 step 0 next 134 2021-07-13 13:39:01.624304: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8d600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152730 step 0 next 135 2021-07-13 13:39:01.624736: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8d700 of size 512 by op SameWorkerRecvDone action_count 9223372089016152731 step 0 next 136 2021-07-13 13:39:01.625171: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8d900 of size 512 by op SameWorkerRecvDone action_count 9223372089016152733 step 0 next 138 2021-07-13 13:39:01.625610: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8db00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152734 step 0 next 139 2021-07-13 13:39:01.626049: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dc8dd00 of size 118016 by op SameWorkerRecvDone action_count 9223372089016152746 step 0 next 80 2021-07-13 13:39:01.626492: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dcaaa00 of size 183808 by op SameWorkerRecvDone action_count 9223372089016152757 step 0 next 81 2021-07-13 13:39:01.626935: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dcd7800 of size 127232 by op SameWorkerRecvDone action_count 9223372089016152759 step 0 next 158 2021-07-13 13:39:01.627373: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dcf6900 of size 156160 by op SameWorkerRecvDone action_count 9223372089016152770 step 0 next 82 2021-07-13 13:39:01.627862: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1cb00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152755 step 0 next 156 2021-07-13 13:39:01.632418: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1cd00 of size 512 by op SameWorkerRecvDone action_count 9223372089016152756 step 0 next 157 2021-07-13 13:39:01.633087: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1cf00 of size 256 by op SameWorkerRecvDone action_count 9223372089016152760 step 0 next 159 2021-07-13 13:39:01.633512: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d000 of size 256 by op SameWorkerRecvDone action_count 9223372089016152761 step 0 next 160 2021-07-13 13:39:01.633974: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d100 of size 256 by op SameWorkerRecvDone action_count 9223372089016152762 step 0 next 161 2021-07-13 13:39:01.634421: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d200 of size 512 by op SameWorkerRecvDone action_count 9223372089016152764 step 0 next 163 2021-07-13 13:39:01.634872: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d400 of size 512 by op SameWorkerRecvDone action_count 9223372089016152765 step 0 next 164 2021-07-13 13:39:01.635307: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152766 step 0 next 165 2021-07-13 13:39:01.635763: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d700 of size 256 by op SameWorkerRecvDone action_count 9223372089016152771 step 0 next 166 2021-07-13 13:39:01.636243: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d800 of size 256 by op SameWorkerRecvDone action_count 9223372089016152772 step 0 next 83 2021-07-13 13:39:01.636675: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd1d900 of size 106752 by op SameWorkerRecvDone action_count 9223372089016152787 step 0 next 179 2021-07-13 13:39:01.637114: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd37a00 of size 114688 by op SameWorkerRecvDone action_count 9223372089016152788 step 0 next 84 2021-07-13 13:39:01.637544: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd53a00 of size 1536 by op SameWorkerRecvDone action_count 9223372089016152677 step 0 next 85 2021-07-13 13:39:01.637977: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd54000 of size 256 by op SameWorkerRecvDone action_count 9223372089016152678 step 0 next 86 2021-07-13 13:39:01.638407: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd54100 of size 256 by op SameWorkerRecvDone action_count 9223372089016152679 step 0 next 87 2021-07-13 13:39:01.638840: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd54200 of size 138240 by op SameWorkerRecvDone action_count 9223372089016152680 step 0 next 88 2021-07-13 13:39:01.639269: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd75e00 of size 1536 by op SameWorkerRecvDone action_count 9223372089016152681 step 0 next 89 2021-07-13 13:39:01.639700: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd76400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152682 step 0 next 90 2021-07-13 13:39:01.640130: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd76500 of size 138240 by op SameWorkerRecvDone action_count 9223372089016152683 step 0 next 91 2021-07-13 13:39:01.640561: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98100 of size 256 by op SameWorkerRecvDone action_count 9223372089016152684 step 0 next 92 2021-07-13 13:39:01.640990: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98200 of size 256 by op SameWorkerRecvDone action_count 9223372089016152685 step 0 next 93 2021-07-13 13:39:01.641422: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152686 step 0 next 94 2021-07-13 13:39:01.641851: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152687 step 0 next 95 2021-07-13 13:39:01.642286: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152688 step 0 next 96 2021-07-13 13:39:01.642715: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152689 step 0 next 97 2021-07-13 13:39:01.643162: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dd98700 of size 138240 by op SameWorkerRecvDone action_count 9223372089016152690 step 0 next 98 2021-07-13 13:39:01.646865: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddba300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152691 step 0 next 99 2021-07-13 13:39:01.647418: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddba400 of size 69120 by op SameWorkerRecvDone action_count 9223372089016152692 step 0 next 100 2021-07-13 13:39:01.647853: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb200 of size 256 by op SameWorkerRecvDone action_count 9223372089016152693 step 0 next 101 2021-07-13 13:39:01.648308: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb300 of size 256 by op SameWorkerRecvDone action_count 9223372089016152694 step 0 next 102 2021-07-13 13:39:01.648757: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152697 step 0 next 103 2021-07-13 13:39:01.649212: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152698 step 0 next 104 2021-07-13 13:39:01.649645: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddcb600 of size 69120 by op SameWorkerRecvDone action_count 9223372089016152699 step 0 next 105 2021-07-13 13:39:01.650080: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dddc400 of size 256 by op SameWorkerRecvDone action_count 9223372089016152700 step 0 next 106 2021-07-13 13:39:01.650512: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dddc500 of size 36864 by op SameWorkerRecvDone action_count 9223372089016152701 step 0 next 107 2021-07-13 13:39:01.650943: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dde5500 of size 256 by op SameWorkerRecvDone action_count 9223372089016152702 step 0 next 108 2021-07-13 13:39:01.651374: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dde5600 of size 256 by op SameWorkerRecvDone action_count 9223372089016152703 step 0 next 109 2021-07-13 13:39:01.651802: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dde5700 of size 36864 by op SameWorkerRecvDone action_count 9223372089016152704 step 0 next 110 2021-07-13 13:39:01.652233: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddee700 of size 256 by op SameWorkerRecvDone action_count 9223372089016152705 step 0 next 111 2021-07-13 13:39:01.652663: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0ddee800 of size 71680 by op SameWorkerRecvDone action_count 9223372089016152706 step 0 next 18446744073709551615 2021-07-13 13:39:01.653096: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 8388608 2021-07-13 13:39:01.653522: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de00000 of size 256 by op SameWorkerRecvDone action_count 9223372089016152707 step 0 next 113 2021-07-13 13:39:01.653952: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de00100 of size 512 by op SameWorkerRecvDone action_count 9223372089016152708 step 0 next 114 2021-07-13 13:39:01.654383: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de00300 of size 221440 by op SameWorkerRecvDone action_count 9223372089016152718 step 0 next 123 2021-07-13 13:39:01.654814: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de36400 of size 221440 by op SameWorkerRecvDone action_count 9223372089016152722 step 0 next 127 2021-07-13 13:39:01.655244: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de6c500 of size 183808 by op SameWorkerRecvDone action_count 9223372089016152723 step 0 next 128 2021-07-13 13:39:01.655679: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0de99300 of size 183808 by op SameWorkerRecvDone action_count 9223372089016152728 step 0 next 133 2021-07-13 13:39:01.656108: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dec6100 of size 283392 by op SameWorkerRecvDone action_count 9223372089016152732 step 0 next 137 2021-07-13 13:39:01.656538: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0df0b400 of size 151808 by op SameWorkerRecvDone action_count 9223372089016152736 step 0 next 140 2021-07-13 13:39:01.656967: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0df30500 of size 183808 by op SameWorkerRecvDone action_count 9223372089016152737 step 0 next 141 2021-07-13 13:39:01.657400: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0df5d300 of size 283392 by op SameWorkerRecvDone action_count 9223372089016152740 step 0 next 144 2021-07-13 13:39:01.657830: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dfa2600 of size 151808 by op SameWorkerRecvDone action_count 9223372089016152744 step 0 next 148 2021-07-13 13:39:01.658265: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0dfc7700 of size 283392 by op SameWorkerRecvDone action_count 9223372089016152752 step 0 next 154 2021-07-13 13:39:01.662505: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e00ca00 of size 221440 by op SameWorkerRecvDone action_count 9223372089016152753 step 0 next 155 2021-07-13 13:39:01.663235: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e042b00 of size 1327104 by op SameWorkerRecvDone action_count 9223372089016152763 step 0 next 162 2021-07-13 13:39:01.663670: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e186b00 of size 1327104 by op SameWorkerRecvDone action_count 9223372089016152773 step 0 next 167 2021-07-13 13:39:01.664126: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e2cab00 of size 106752 by op SameWorkerRecvDone action_count 9223372089016152779 step 0 next 172 2021-07-13 13:39:01.664574: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e2e4c00 of size 1327104 by op SameWorkerRecvDone action_count 9223372089016152782 step 0 next 175 2021-07-13 13:39:01.665029: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e428c00 of size 89856 by op SameWorkerRecvDone action_count 9223372089016152794 step 0 next 185 2021-07-13 13:39:01.665466: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b0e43eb00 of size 1840384 by op UNUSED action_count 0 step 0 next 18446744073709551615 2021-07-13 13:39:01.665901: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 1073741824 2021-07-13 13:39:01.666336: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0e600000 of size 586641152 by op SameWorkerRecvDone action_count 9223372089016152802 step 0 next 188 2021-07-13 13:39:01.666768: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b31576f00 of size 146660352 by op SameWorkerRecvDone action_count 9223372089016152803 step 0 next 189 2021-07-13 13:39:01.667210: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at b3a154b00 of size 340440320 by op UNUSED action_count 0 step 0 next 18446744073709551615 2021-07-13 13:39:01.667639: I tensorflow/core/common_runtime/bfc_allocator.cc:1051] Summary of in-use Chunks by size: 2021-07-13 13:39:01.668080: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 87 Chunks of size 256 totalling 21.8KiB 2021-07-13 13:39:01.668502: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 40 Chunks of size 512 totalling 20.0KiB 2021-07-13 13:39:01.668930: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1280 totalling 1.2KiB 2021-07-13 13:39:01.669357: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 1536 totalling 7.5KiB 2021-07-13 13:39:01.669785: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 8 Chunks of size 3584 totalling 28.0KiB 2021-07-13 13:39:01.670214: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 36864 totalling 108.0KiB 2021-07-13 13:39:01.670641: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 69120 totalling 270.0KiB 2021-07-13 13:39:01.671074: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 71680 totalling 70.0KiB 2021-07-13 13:39:01.671501: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 89856 totalling 175.5KiB 2021-07-13 13:39:01.671929: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 106752 totalling 312.8KiB 2021-07-13 13:39:01.672362: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 114688 totalling 112.0KiB 2021-07-13 13:39:01.672789: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 118016 totalling 115.2KiB 2021-07-13 13:39:01.673219: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 127232 totalling 372.8KiB 2021-07-13 13:39:01.673645: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 138240 totalling 540.0KiB 2021-07-13 13:39:01.674077: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 151808 totalling 444.8KiB 2021-07-13 13:39:01.678182: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 156160 totalling 152.5KiB 2021-07-13 13:39:01.679297: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 183808 totalling 897.5KiB 2021-07-13 13:39:01.679733: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 221440 totalling 865.0KiB 2021-07-13 13:39:01.680204: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 229120 totalling 223.8KiB 2021-07-13 13:39:01.680680: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 283392 totalling 1.08MiB 2021-07-13 13:39:01.681195: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 1327104 totalling 5.06MiB 2021-07-13 13:39:01.681690: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 146660352 totalling 139.87MiB 2021-07-13 13:39:01.682223: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 586641152 totalling 559.46MiB 2021-07-13 13:39:01.682646: I tensorflow/core/common_runtime/bfc_allocator.cc:1058] Sum Total of in-use chunks: 710.10MiB 2021-07-13 13:39:01.683099: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] total_region_allocatedbytes: 1088421888 memorylimit: 6292504576 available bytes: 5204082688 curr_region_allocationbytes: 1073741824 2021-07-13 13:39:01.683561: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Stats: Limit: 6292504576 InUse: 744595456 MaxInUse: 744595456 NumAllocs: 200 MaxAllocSize: 586641152 Reserved: 0 PeakReserved: 0 LargestFreeBlock: 0
2021-07-13 13:39:01.684082: W tensorflow/core/common_runtime/bfc_allocator.cc:467] *****___ 2021-07-13 13:39:01.684564: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_ops.cc:662 : Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call return fn(*args) File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1359, in _run_fn return self._call_tf_sessionrun(options, feed_dict, fetch_list, File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1451, in _call_tf_sessionrun return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict, tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node CNN1/CNN1_conv}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node CNN1/CNN1_conv}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations. 0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Code\MAI\Source\Services\DCSCN\sr.py", line 51, in
tf.app.run()
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\absl\app.py", line 312, in run
_run_main(main, args)
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\absl\app.py", line 258, in _run_main
sys.exit(main(argv))
File "C:\Code\MAI\Source\Services\DCSCN\sr.py", line 47, in main
model.do_for_file(FLAGS.file, FLAGS.output_dir)
File "C:\Code\MAI\Source\Services\DCSCN\DCSCN.py", line 618, in do_for_file
output_y_image = self.do(input_yimage)
File "C:\Code\MAI\Source\Services\DCSCN\DCSCN.py", line 580, in do
y = self.sess.run(self.y, feed_dict={self.x: image.reshape(1, image.shape[0], image.shape[1], ch),
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1368, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "C:\Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node CNN1/CNN1_conv (defined at Code\MAI\Source\Services\DCSCN\helper\tf_graph.py:105) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[1,4586,7995,96] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node CNN1/CNN1_conv (defined at Code\MAI\Source\Services\DCSCN\helper\tf_graph.py:105) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations. 0 derived errors ignored.
Errors may have originated from an input operation. Input Source operations connected to node CNN1/CNN1_conv: x (defined at Code\MAI\Source\Services\DCSCN\DCSCN.py:227)
Input Source operations connected to node CNN1/CNN1_conv: x (defined at Code\MAI\Source\Services\DCSCN\DCSCN.py:227)
Original stack trace for 'CNN1/CNN1_conv': File "Code\MAI\Source\Services\DCSCN\sr.py", line 51, in
tf.app.run()
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\absl\app.py", line 312, in run
_run_main(main, args)
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\absl\app.py", line 258, in _run_main
sys.exit(main(argv))
File "Code\MAI\Source\Services\DCSCN\sr.py", line 40, in main
model.build_graph()
File "Code\MAI\Source\Services\DCSCN\DCSCN.py", line 255, in build_graph
self.build_conv("CNN%d" % (i + 1), input_tensor, self.cnn_size, input_feature_num,
File "Code\MAI\Source\Services\DCSCN\helper\tf_graph.py", line 124, in build_conv
h = self.conv2d(input_tensor, w, self.cnn_stride, bias=b, use_batch_norm=use_batch_norm, name=name)
File "Code\MAI\Source\Services\DCSCN\helper\tf_graph.py", line 105, in conv2d
output = tf.nn.conv2d(input_tensor, w, strides=[1, stride, stride, 1], padding="SAME", name=name + "_conv")
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper
return target(*args, **kwargs)
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 2269, in conv2d
return gen_nn_ops.conv2d(
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\ops\gen_nnops.py", line 968, in conv2d
, _, _op, _outputs = _op_def_library._apply_op_helper(
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 748, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\framework\ops.py", line 3557, in _create_op_internal
ret = Operation(
File "Users\Justin.\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\framework\ops.py", line 2045, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)