Dean-TianZhang
commented
7 years ago @DeepStillWater NameError: name 'xrange' is not defined, you should change 'xrange' to 'range'
Open DeepStillWater opened 7 years ago
Dean-TianZhang
commented
7 years ago @DeepStillWater NameError: name 'xrange' is not defined, you should change 'xrange' to 'range'
DeepStillWater
commented
7 years ago @Dean-TianZhang Thanks,it did solve it,but new issues occured like this:
iter 333: image-id:0577807, time:0.526(sec), regular_loss: 0.254317, total-loss 19605270528.0000(352860800.0000, 19252410368.0000, 0.000000, 0.0000, 0.0000), instances: 2, batch:(2|34, 0|2, 0|0)
iter 334: image-id:0184626, time:0.944(sec), regular_loss: 86662172377088.000000, total-loss nan(0.0809, 0.4228, nan, 0.0000, 0.0000), instances: 29, batch:(43|190, 0|64, 0|0)
[[ 300.64624023 436.44924927 343.46664429 555.89666748 1. ]
[ 807.95678711 382.88528442 817.14355469 403.21923828 1. ]
[ 758.04443359 375.60119629 769.94116211 401.33575439 1. ]
[ 181.50631714 134.76516724 215.21684265 169.55195618 34. ]
[ 467.68048096 221.11712646 494.95257568 252.67507935 34. ]
[ 578.69073486 281.65768433 594.23907471 302.83724976 34. ]
[ 343.19763184 242.98860168 355.32495117 255.59642029 34. ]
[ 320.61505127 210.66186523 336.12496948 227.92071533 34. ]
[ 820.83361816 367.72134399 827.11828613 386.38317871 1. ]
[ 625.95074463 361.18676758 630.19818115 374.37115479 1. ]
[ 404.43002319 215.10150146 426.39758301 246.7555542 34. ]
[ 24.38919449 242.4888916 44.53095627 262.64987183 34. ]
[ 195.09431458 255.01983643 211.46907043 268.72314453 34. ]
[ 296.18737793 486.15014648 314.253479 531.12316895 38. ]
[ 582.4576416 366.37600708 597.94836426 380.1946106 1. ]
[ 549.28527832 360.61022949 553.35974121 374.98620605 1. ]
[ 354.7098999 362.72433472 360.12976074 370.14294434 1. ]
[ 400.79763794 354.55615234 406.85165405 366.222229 1. ]
[ 878.68353271 368.02883911 884.08410645 382.05886841 1. ]
[ 888.0625 370.21984863 897.53753662 388.93933105 1. ]
[ 454.09246826 228.36276245 475.07989502 248.73513794 34. ]
[ 729.94592285 245.92913818 744.57177734 270.68347168 34. ]
[ 589.2612915 298.89730835 595.18078613 307.69970703 34. ]
[ 283.48348999 248.63903809 307.46908569 269.97238159 34. ]
[ 407.44744873 359.72613525 411.81021118 369.08587646 1. ]
[ 351.57717896 203.20481873 385.26846313 239.74053955 34. ]
[ 739.15197754 368.18261719 745.57116699 379.11831665 1. ]
[ 457.41741943 344.02401733 882.16217041 392.07208252 1. ]
[ 61.50150299 209.48950195 891.77178955 336.33633423 34. ]]
Traceback (most recent call last):
File "train/train.py", line 222, in
kevinkit
commented
7 years ago Change the line "raise" to "raise exception"
DeepStillWater
commented
7 years ago I'm sorry to disturb you,I follow your steps,the error occurred again @kevinkit
2017-05-26 21:23:30.535818: W tensorflow/core/common_runtime/bfc_allocator.cc:277] **** 2017-05-26 21:23:30.535845: W tensorflow/core/framework/op_kernel.cc:1152] Resource exhausted: OOM when allocating tensor with shape[1,256,40,193] Traceback (most recent call last): File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1039, in _do_call return fn(*args) File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1021, in _run_fn status, run_metadata) File "/opt/anaconda3/lib/python3.5/contextlib.py", line 66, in exit next(self.gen) File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,256,40,193] [[Node: resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/Relu, resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/weights/read)]] [[Node: pyramid_2/strided_slice_37/_1033 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_6143_pyramid_2/strided_slice_37", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train/train.py", line 222, in
Caused by op 'resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/convolution', defined at:
File "train/train.py", line 222, in
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,256,40,193] [[Node: resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/Relu, resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/weights/read)]] [[Node: pyramid_2/strided_slice_37/_1033 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_6143_pyramid_2/strided_slice_37", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
kevinkit
commented
7 years ago You ran out of Memory on your gpu , check with nvidia-smi if you have enough Memory available or if any other program is running on the gpu atm
mattdingmeng
commented
7 years ago @DeepStillWater When I run this training code, I also have the out of memory problem on my GPU. I am curious that are you studying/working at OSU?
DeepStillWater
commented
7 years ago @mattdingmeng No,I'm just studying in a university of Chinese.
opencvfun
commented
7 years ago @DeepStillWater Thanks for your help ! I have also met this problem of "Out of memory" .. Could you tell me How much GPU memory I need ? I am using GTX 970 only with 4GB GPU memory ...
DeepStillWater
commented
7 years ago @opencvfun I am afraid 4GB memory is not enough，may be you can change the nums of picture every iters .
kevinkit
commented
7 years ago The original paper hat 4 x 8 gb available at a Batchsize of 2, so assuming that the images / computation does not really get split up among the GPUs, a Batchsize of 1 could maybe help to only need 4 GB - but like @DeepStillWater said, 4gb may not be enough here, due to some overhead.
However, you can check with the nvidia-smi command (assuming you are using any unix distr) you can check wether other processes are currently using the GPU and kill them to free up some mem.
2017-05-25 13:23:12.879749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 2017-05-25 13:23:12.879754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y 2017-05-25 13:23:12.879760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:06:00.0) --restore_previous_if_exists is set, but failed to restore in ./output/mask_rcnn/ None restoring resnet_v1_50/conv1/weights:0 restoring resnet_v1_50/conv1/BatchNorm/beta:0 restoring resnet_v1_50/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/weights:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/weights:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/weights:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/weights:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/weights:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/weights:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/weights:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/beta:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean:0 restoring resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance:0 restoring resnet_v1_50/logits/weights:0 restoring resnet_v1_50/logits/biases:0 Restored 267(544) vars from ./data/pretrained_models/resnet_v1_50.ckpt Traceback (most recent call last): Traceback (most recent call last): File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 82, in call File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 82, in call ret = func(args) File "train/../libs/boxes/anchor.py", line 23, in anchors_plane ret = func(args) File "train/../libs/boxes/anchor.py", line 23, in anchors_plane anc = anchors(scales, ratios, base) File "train/../libs/boxes/anchor.py", line 10, in anchors anc = anchors(scales, ratios, base) File "train/../libs/boxes/anchor.py", line 10, in anchors return generate_anchors(base_size=base, scales=np.asarray(scales, np.int32), ratios=ratios) File "train/../libs/boxes/anchor.py", line 38, in generate_anchors return generate_anchors(base_size=base, scales=np.asarray(scales, np.int32), ratios=ratios) File "train/../libs/boxes/anchor.py", line 38, in generate_anchors for i in xrange(ratio_anchors.shape[0])]) NameError: name 'xrange' is not defined 2017-05-25 13:25:32.432683: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log. for i in xrange(ratio_anchors.shape[0])]) NameError: name 'xrange' is not defined 2017-05-25 13:25:32.432796: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. 2017-05-25 13:25:32.456233: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.456326: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.456379: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.456397: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.456511: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555291: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555329: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555346: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555377: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555393: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555412: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555525: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555617: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555638: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555699: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555717: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555732: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555746: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555765: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555780: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555803: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555879: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555923: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.555940: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.556028: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.556048: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.556208: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] 2017-05-25 13:25:32.556247: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]] Traceback (most recent call last): File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1039, in _do_call return fn(*args) File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1021, in _run_fn status, run_metadata) File "/opt/anaconda3/lib/python3.5/contextlib.py", line 66, in exit next(self.gen) File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.InternalError: Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "train/train.py", line 222, in
train()
File "train/train.py", line 190, in train
batch_info )
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 778, in run
run_metadata_ptr)
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Failed to run py callback pyfunc_0: see error log.
[[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]]
Caused by op 'pyramid_1/GenAnchors/PyFunc', defined at: File "train/train.py", line 222, in
train()
File "train/train.py", line 136, in train
loss_weights=[0.2, 0.2, 1.0, 0.2, 1.0])
File "train/../libs/nets/pyramid_network.py", line 526, in build
is_training=is_training, gt_boxes=gt_boxes)
File "train/../libs/nets/pyramid_network.py", line 225, in build_heads
all_anchors = gen_all_anchors(height, width, stride, anchor_scales)
File "train/../libs/layers/wrapper.py", line 149, in gen_all_anchors
[tf.float64]
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 189, in py_func
input=inp, token=token, Tout=Tout, name=name)
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/gen_script_ops.py", line 40, in _py_func
name=name)
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/opt/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1228, in init
self._traceback = _extract_stack()
InternalError (see above for traceback): Failed to run py callback pyfunc_0: see error log. [[Node: pyramid_1/GenAnchors/PyFunc = PyFunc[Tin=[DT_INT32, DT_INT32, DT_INT32, DT_INT32], Tout=[DT_DOUBLE], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](pyramid_1/strided_slice/_847, pyramid_1/strided_slice_1/_849, pyramid_1/GenAnchors/PyFunc/input_2, pyramid_1/GenAnchors/PyFunc/input_3)]]
The version of my python is 3.5.2 and tensorflow is 1.1.0,how can I solve it?