search

InnerPeace-Wu / densecap-tensorflow

Re-implement CVPR2017 paper: "dense captioning with joint inference and visual context" and minor changes in Tensorflow. (mAP 8.296 after 500k iters of training)
MIT License
61 stars 28 forks source link

使用resnet_v2作为基本网络，出现InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [1024] rhs shape= [2048] #19

Closed Dontfall closed 6 years ago

Dontfall commented 6 years ago

@InnerPeace-Wu 使用resnet_v2作为基本网络，在训练时出现如下错误，希望能得到你的建议，非常感谢！！！

:~/densecap-tensorflow$ bash scripts/dense_cap_train.sh visual_genome_1.2 resnet_v2_50 /home/xiewei/densecap-tensorflow/slim_models/resnet_v2_50.ckpt /home/xiewei/densecap-tensorflow/visual_genome 1

Caused by op u'save_1/Assign_217', defined at: File "./tools/train_net.py", line 221, in main() File "./tools/train_net.py", line 217, in main max_iters=args.max_iters) File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 485, in train_net sw.train_model(sess, max_iters) File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 325, in train_model rate, last_snapshot_iter, stepsizes, np_paths, ss_paths = self.initialize(sess) File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 236, in initialize restorer = tf.train.Saver(variables_to_restore) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1218, in init self.build() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1227, in build self._build(self._filename, build_save=True, build_restore=True) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1263, in _build build_save=build_save, build_restore=build_restore) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 751, in _build_internal restore_sequentially, reshape) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 439, in _AddRestoreOps assign_ops.append(saveable.restore(tensors, shapes)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 160, in restore self.op.get_shape().is_fully_defined()) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 276, in assign validate_shape=validate_shape) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 57, in assign use_locking=use_locking, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1470, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [1024] rhs shape= [2048] [[Node: save_1/Assign_217 = Assign[T=DT_FLOAT, _class=["loc:@resnet_v2_50/postnorm/moving_variance"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](resnet_v2_50/postnorm/moving_variance, save_1/RestoreV2_217/_9)]]

Dontfall commented 6 years ago

@InnerPeace-Wu 恳求大佬回复，万分感谢！！！

InnerPeace-Wu commented 6 years ago

不好意思，最近比较忙。明天我再好好看一下。 best,

InnerPeace-Wu commented 6 years ago
  1. 你可以把postnorm这个节点的输出维度打印出来看是否正确，报错的信息就是指里的restore的时候与Imagenet train之后 resnetv2维数不对应。注意输入图片的size大小。
  2. 最简单的解决方式是把postnorm从restore_checkpoint中排除。但不推荐这种方式，没有解决实质问题。

建议把graph加到tensorborad中进行debug，这部分我好像没有写，需要自己探索一下。 best,

Dontfall commented 6 years ago

@InnerPeace-Wu 感谢你的建议，谢谢，我会尽量去试试。

Dontfall commented 6 years ago

@InnerPeace-Wu 谢谢你的建议，我已经解决了问题。