Open alanxuefei opened 3 years ago
Encounter the following error. Would you like to share how to fix it? Thanks in advance
(base) alan@alan-Precision-5820-Tower:~/Desktop/morefusion/examples/ycb_video/instance_segm$ mpirun -n 0 python train_multi.py / True /bbox_head True /bbox_head/conf True /bbox_head/fc1 True /bbox_head/fc2 True /bbox_head/loc True /extractor False /extractor/base False /extractor/base/conv1 False /extractor/base/conv1/bn False /extractor/base/conv1/conv False /extractor/base/res2 False /extractor/base/res2/a False /extractor/base/res2/a/conv1 False /extractor/base/res2/a/conv1/bn False /extractor/base/res2/a/conv1/conv False /extractor/base/res2/a/conv2 False /extractor/base/res2/a/conv2/bn False /extractor/base/res2/a/conv2/conv False /extractor/base/res2/a/conv3 False /extractor/base/res2/a/conv3/bn False /extractor/base/res2/a/conv3/conv False /extractor/base/res2/a/residual_conv False /extractor/base/res2/a/residual_conv/bn False /extractor/base/res2/a/residual_conv/conv False /extractor/base/res2/b1 False /extractor/base/res2/b1/conv1 False /extractor/base/res2/b1/conv1/bn False /extractor/base/res2/b1/conv1/conv False /extractor/base/res2/b1/conv2 False /extractor/base/res2/b1/conv2/bn False /extractor/base/res2/b1/conv2/conv False /extractor/base/res2/b1/conv3 False /extractor/base/res2/b1/conv3/bn False /extractor/base/res2/b1/conv3/conv False /extractor/base/res2/b2 False /extractor/base/res2/b2/conv1 False /extractor/base/res2/b2/conv1/bn False /extractor/base/res2/b2/conv1/conv False /extractor/base/res2/b2/conv2 False /extractor/base/res2/b2/conv2/bn False /extractor/base/res2/b2/conv2/conv False /extractor/base/res2/b2/conv3 False /extractor/base/res2/b2/conv3/bn False /extractor/base/res2/b2/conv3/conv False /extractor/base/res3 False /extractor/base/res3/a False /extractor/base/res3/a/conv1 False /extractor/base/res3/a/conv1/bn False /extractor/base/res3/a/conv1/conv False /extractor/base/res3/a/conv2 False /extractor/base/res3/a/conv2/bn False /extractor/base/res3/a/conv2/conv False /extractor/base/res3/a/conv3 False /extractor/base/res3/a/conv3/bn False /extractor/base/res3/a/conv3/conv False /extractor/base/res3/a/residual_conv False /extractor/base/res3/a/residual_conv/bn False /extractor/base/res3/a/residual_conv/conv False /extractor/base/res3/b1 False /extractor/base/res3/b1/conv1 False /extractor/base/res3/b1/conv1/bn False /extractor/base/res3/b1/conv1/conv False /extractor/base/res3/b1/conv2 False /extractor/base/res3/b1/conv2/bn False /extractor/base/res3/b1/conv2/conv False /extractor/base/res3/b1/conv3 False /extractor/base/res3/b1/conv3/bn False /extractor/base/res3/b1/conv3/conv False /extractor/base/res3/b2 False /extractor/base/res3/b2/conv1 False /extractor/base/res3/b2/conv1/bn False /extractor/base/res3/b2/conv1/conv False /extractor/base/res3/b2/conv2 False /extractor/base/res3/b2/conv2/bn False /extractor/base/res3/b2/conv2/conv False /extractor/base/res3/b2/conv3 False /extractor/base/res3/b2/conv3/bn False /extractor/base/res3/b2/conv3/conv False /extractor/base/res3/b3 False /extractor/base/res3/b3/conv1 False /extractor/base/res3/b3/conv1/bn False /extractor/base/res3/b3/conv1/conv False /extractor/base/res3/b3/conv2 False /extractor/base/res3/b3/conv2/bn False /extractor/base/res3/b3/conv2/conv False /extractor/base/res3/b3/conv3 False /extractor/base/res3/b3/conv3/bn False /extractor/base/res3/b3/conv3/conv False /extractor/base/res4 False /extractor/base/res4/a False /extractor/base/res4/a/conv1 False /extractor/base/res4/a/conv1/bn False /extractor/base/res4/a/conv1/conv False /extractor/base/res4/a/conv2 False /extractor/base/res4/a/conv2/bn False /extractor/base/res4/a/conv2/conv False /extractor/base/res4/a/conv3 False /extractor/base/res4/a/conv3/bn False /extractor/base/res4/a/conv3/conv False /extractor/base/res4/a/residual_conv False /extractor/base/res4/a/residual_conv/bn False /extractor/base/res4/a/residual_conv/conv False /extractor/base/res4/b1 False /extractor/base/res4/b1/conv1 False /extractor/base/res4/b1/conv1/bn False /extractor/base/res4/b1/conv1/conv False /extractor/base/res4/b1/conv2 False /extractor/base/res4/b1/conv2/bn False /extractor/base/res4/b1/conv2/conv False /extractor/base/res4/b1/conv3 False /extractor/base/res4/b1/conv3/bn False /extractor/base/res4/b1/conv3/conv False /extractor/base/res4/b2 False /extractor/base/res4/b2/conv1 False /extractor/base/res4/b2/conv1/bn False /extractor/base/res4/b2/conv1/conv False /extractor/base/res4/b2/conv2 False /extractor/base/res4/b2/conv2/bn False /extractor/base/res4/b2/conv2/conv False /extractor/base/res4/b2/conv3 False /extractor/base/res4/b2/conv3/bn False /extractor/base/res4/b2/conv3/conv False /extractor/base/res4/b3 False /extractor/base/res4/b3/conv1 False /extractor/base/res4/b3/conv1/bn False /extractor/base/res4/b3/conv1/conv False /extractor/base/res4/b3/conv2 False /extractor/base/res4/b3/conv2/bn False /extractor/base/res4/b3/conv2/conv False /extractor/base/res4/b3/conv3 False /extractor/base/res4/b3/conv3/bn False /extractor/base/res4/b3/conv3/conv False /extractor/base/res4/b4 False /extractor/base/res4/b4/conv1 False /extractor/base/res4/b4/conv1/bn False /extractor/base/res4/b4/conv1/conv False /extractor/base/res4/b4/conv2 False /extractor/base/res4/b4/conv2/bn False /extractor/base/res4/b4/conv2/conv False /extractor/base/res4/b4/conv3 False /extractor/base/res4/b4/conv3/bn False /extractor/base/res4/b4/conv3/conv False /extractor/base/res4/b5 False /extractor/base/res4/b5/conv1 False /extractor/base/res4/b5/conv1/bn False /extractor/base/res4/b5/conv1/conv False /extractor/base/res4/b5/conv2 False /extractor/base/res4/b5/conv2/bn False /extractor/base/res4/b5/conv2/conv False /extractor/base/res4/b5/conv3 False /extractor/base/res4/b5/conv3/bn False /extractor/base/res4/b5/conv3/conv False /extractor/base/res5 False /extractor/base/res5/a False /extractor/base/res5/a/conv1 False /extractor/base/res5/a/conv1/bn False /extractor/base/res5/a/conv1/conv False /extractor/base/res5/a/conv2 False /extractor/base/res5/a/conv2/bn False /extractor/base/res5/a/conv2/conv False /extractor/base/res5/a/conv3 False /extractor/base/res5/a/conv3/bn False /extractor/base/res5/a/conv3/conv False /extractor/base/res5/a/residual_conv False /extractor/base/res5/a/residual_conv/bn False /extractor/base/res5/a/residual_conv/conv False /extractor/base/res5/b1 False /extractor/base/res5/b1/conv1 False /extractor/base/res5/b1/conv1/bn False /extractor/base/res5/b1/conv1/conv False /extractor/base/res5/b1/conv2 False /extractor/base/res5/b1/conv2/bn False /extractor/base/res5/b1/conv2/conv False /extractor/base/res5/b1/conv3 False /extractor/base/res5/b1/conv3/bn False /extractor/base/res5/b1/conv3/conv False /extractor/base/res5/b2 False /extractor/base/res5/b2/conv1 False /extractor/base/res5/b2/conv1/bn False /extractor/base/res5/b2/conv1/conv False /extractor/base/res5/b2/conv2 False /extractor/base/res5/b2/conv2/bn False /extractor/base/res5/b2/conv2/conv False /extractor/base/res5/b2/conv3 False /extractor/base/res5/b2/conv3/bn False /extractor/base/res5/b2/conv3/conv False /extractor/inner False /extractor/inner/0 False /extractor/inner/1 False /extractor/inner/2 False /extractor/inner/3 False /extractor/outer False /extractor/outer/0 False /extractor/outer/1 False /extractor/outer/2 False /extractor/outer/3 False /mask_head True /mask_head/conv1 True /mask_head/conv1/conv True /mask_head/conv2 True /mask_head/conv2/conv True /mask_head/conv3 True /mask_head/conv3/conv True /mask_head/conv4 True /mask_head/conv4/conv True /mask_head/conv5 True /mask_head/seg True /rpn False /rpn/conf False /rpn/conv False /rpn/loc False /home/alan/Desktop/morefusion/examples/ycb_video/instance_segm/transforms.py:86: DeprecationWarning: AffineCv2 is deprecated. Use imgaug.augmenters.geometric.Affine(..., backend='cv2') instead. shear=(-15, 15), /home/alan/Desktop/morefusion/examples/ycb_video/instance_segm/transforms.py:86: DeprecationWarning: AffineCv2 is deprecated. Use imgaug.augmenters.geometric.Affine(..., backend='cv2') instead. shear=(-15, 15), /home/alan/Desktop/morefusion/examples/ycb_video/instance_segm/transforms.py:86: DeprecationWarning: AffineCv2 is deprecated. Use imgaug.augmenters.geometric.Affine(..., backend='cv2') instead. shear=(-15, 15), /home/alan/Desktop/morefusion/examples/ycb_video/instance_segm/transforms.py:86: DeprecationWarning: AffineCv2 is deprecated. Use imgaug.augmenters.geometric.Affine(..., backend='cv2') instead. shear=(-15, 15), /home/alan/Desktop/morefusion/examples/ycb_video/instance_segm/transforms.py:86: DeprecationWarning: AffineCv2 is deprecated. Use imgaug.augmenters.geometric.Affine(..., backend='cv2') instead. shear=(-15, 15), /home/alan/Desktop/morefusion/examples/ycb_video/instance_segm/transforms.py:86: DeprecationWarning: AffineCv2 is deprecated. Use imgaug.augmenters.geometric.Affine(..., backend='cv2') instead. shear=(-15, 15), /home/alan/Desktop/morefusion/examples/ycb_video/instance_segm/transforms.py:86: DeprecationWarning: AffineCv2 is deprecated. Use imgaug.augmenters.geometric.Affine(..., backend='cv2') instead. shear=(-15, 15), /home/alan/Desktop/morefusion/examples/ycb_video/instance_segm/transforms.py:86: DeprecationWarning: AffineCv2 is deprecated. Use imgaug.augmenters.geometric.Affine(..., backend='cv2') instead. shear=(-15, 15), Exception in main training loop: Invalid operation is performed in: LinearFunction (Forward) Expect: b.shape[0] == W.shape[0] Actual: 88 != 324 Traceback (most recent call last): File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/training/trainer.py", line 343, in run update() File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/training/updaters/standard_updater.py", line 240, in update self.update_core() File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/training/updaters/standard_updater.py", line 253, in update_core optimizer.update(loss_func, *in_arrays) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainermn/optimizers.py", line 21, in update loss = lossfun(*args, **kwds) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/link.py", line 287, in __call__ out = forward(*args, **kwargs) File "train_multi.py", line 104, in forward head_locs, head_confs = self.model.bbox_head(hs, rois, roi_indices) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/link.py", line 287, in __call__ out = forward(*args, **kwargs) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainercv/links/model/fpn/bbox_head.py", line 92, in forward locs = self.loc(h) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/link.py", line 287, in __call__ out = forward(*args, **kwargs) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/links/connection/linear.py", line 183, in forward return linear.linear(x, self.W, self.b, n_batch_axes=n_batch_axes) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/functions/connection/linear.py", line 308, in linear y, = LinearFunction().apply(args) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/function_node.py", line 307, in apply self._check_data_type_forward(in_data) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/function_node.py", line 455, in _check_data_type_forward self.check_type_forward(in_type) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/functions/connection/linear.py", line 37, in check_type_forward b_type.shape[0] == w_type.shape[0], File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/utils/type_check.py", line 564, in expect expr.expect() File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/utils/type_check.py", line 497, in expect '{0} {1} {2}'.format(left, self.inv, right)) Will finalize trainer extensions and updater before reraising the exception. Traceback (most recent call last): File "train_multi.py", line 376, in <module> main() File "train_multi.py", line 372, in main trainer.run() File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/training/trainer.py", line 376, in run six.reraise(*exc_info) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/training/trainer.py", line 343, in run update() File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/training/updaters/standard_updater.py", line 240, in update self.update_core() File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/training/updaters/standard_updater.py", line 253, in update_core optimizer.update(loss_func, *in_arrays) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainermn/optimizers.py", line 21, in update loss = lossfun(*args, **kwds) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/link.py", line 287, in __call__ out = forward(*args, **kwargs) File "train_multi.py", line 104, in forward head_locs, head_confs = self.model.bbox_head(hs, rois, roi_indices) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/link.py", line 287, in __call__ out = forward(*args, **kwargs) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainercv/links/model/fpn/bbox_head.py", line 92, in forward locs = self.loc(h) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/link.py", line 287, in __call__ out = forward(*args, **kwargs) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/links/connection/linear.py", line 183, in forward return linear.linear(x, self.W, self.b, n_batch_axes=n_batch_axes) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/functions/connection/linear.py", line 308, in linear y, = LinearFunction().apply(args) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/function_node.py", line 307, in apply self._check_data_type_forward(in_data) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/function_node.py", line 455, in _check_data_type_forward self.check_type_forward(in_type) File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/functions/connection/linear.py", line 37, in check_type_forward b_type.shape[0] == w_type.shape[0], File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/utils/type_check.py", line 564, in expect expr.expect() File "/home/alan/Desktop/morefusion/.anaconda3/lib/python3.7/site-packages/chainer/utils/type_check.py", line 497, in expect '{0} {1} {2}'.format(left, self.inv, right)) chainer.utils.type_check.InvalidType: Invalid operation is performed in: LinearFunction (Forward) Expect: b.shape[0] == W.shape[0] Actual: 88 != 324
If batchsize = 2 as below, it works.
parser.add_argument("--batchsize", type=int, default=2, help="batch size")
Encounter the following error. Would you like to share how to fix it? Thanks in advance