a-maci
commented
7 years ago I updated to TF 1.0.0. Getting a different error this time.
Traceback (most recent call last):
File "tpInception_resnet_v2.py", line 204, in <module>
SyncMultiGPUTrainer(config).train()
File "/home/akm/TF/tpNew/tensorpack/tensorpack/train/base.py", line 64, in train
self.setup()
File "/home/akm/TF/tpNew/tensorpack/tensorpack/train/base.py", line 130, in setup
self._setup() # subclass will setup the graph
File "/home/akm/TF/tpNew/tensorpack/tensorpack/train/multigpu.py", line 133, in _setup
self.config.tower, lambda: self._get_cost_and_grad()[1])
File "/home/akm/TF/tpNew/tensorpack/tensorpack/train/multigpu.py", line 41, in _multi_tower_grads
grad_list.append(get_tower_grad_func())
File "/home/akm/TF/tpNew/tensorpack/tensorpack/train/multigpu.py", line 133, in <lambda>
self.config.tower, lambda: self._get_cost_and_grad()[1])
File "/home/akm/TF/tpNew/tensorpack/tensorpack/train/feedfree.py", line 54, in _get_cost_and_grad
self.build_train_tower()
File "/home/akm/TF/tpNew/tensorpack/tensorpack/train/feedfree.py", line 43, in build_train_tower
f()
File "/home/akm/TF/tpNew/tensorpack/tensorpack/train/feedfree.py", line 36, in f
self.model.build_graph(inputs)
File "/home/akm/TF/tpNew/tensorpack/tensorpack/models/model_desc.py", line 113, in build_graph
self._build_graph(model_inputs)
File "tpInception_resnet_v2.py", line 53, in _build_graph
logits, end_points = inception_resnet_v2(image, is_training=is_training)
File "/home/akm/TF/tpNew/tensorpack/examples/IRv2/inception_resnet_v2.py", line 169, in inception_resnet_v2
net = tf.concat(3, [tower_conv, tower_conv1_1, tower_conv2_2, tower_pool_1])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1047, in concat
dtype=dtypes.int32).get_shape(
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 651, in convert_to_tensor
as_ref=False)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 716, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 165, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 367, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 302, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).__name__))
TypeError: Expected int32, got list containing Tensors of type '_Message' instead.
Prefetch process exited.
Prefetch process exited.I guess something to do with the concat API. I did minor mod to the file.
ppwwyyxx
Not an issue specifically with tensorpack.
I was trying to load a checkpoint model for Inception-Resnet-v2 (uploaded here). I re-used the tf.slim model for this network mentioned here and use tensorpack as the front-end. Attached are both these file (in text format since I cant upload .py files).
I am running into restoring the graph variables problems. Here is the last portion of the error log:
This error is strange since I dont have a tensor called global_step. I even tried to change the dtype of GLOBAL_STEP_OP_NAME in tfutils/common.py to tf.int64 but I get some other error with it.
Any suggestions/pointer as to what I am doing wrong and/or how to fix this? BTW, I am still using tf version 0.12.0rc1. Not sure if going to 1.0.0 could fix this problem.
run command: python tpInception_resnet_v2.py --gpu 0,1,2,3,4,5,6,7 --data /tank/imagenet-tensorpack-data --load googleModel/inception_resnet_v2_2016_08_30.ckpt