Incompatible shapes error

[lexical]

Hi, I am trying to train OpenImages V4 with 600 classes. The training stopped with the following error. Wonder if this error is from Luminoth. Any suggestion to get this fixed?

INFO:tensorflow:step: 1986, file: 0012c270e7a0d8e9, train_loss: 7.52168273926, in 15.08s
Traceback (most recent call last):
  File "/root/venv2/bin/lumi", line 11, in <module>
    sys.exit(cli())
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/root/venv2/local/lib/python2.7/site-packages/luminoth/train.py", line 307, in train
    config, environment=environment
  File "/root/venv2/local/lib/python2.7/site-packages/luminoth/train.py", line 239, in run
    ], options=run_options)  
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1148, in run
    run_metadata=run_metadata)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1239, in run
    raise six.reraise(*original_exc_info)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1224, in run
    return self._sess.run(*args, **kwargs)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1296, in run
    run_metadata=run_metadata)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1076, in run
    return self._sess.run(*args, **kwargs)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1110, in _run
    feed_dict_tensor, options, run_metadata)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1286, in _do_run
    run_metadata)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1308, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [3,4] vs. [9,4]
         [[{{node losses/RCNNLoss/sub_1}} = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](losses/RCNNLos
atherV2)]]

Caused by op u'losses/RCNNLoss/sub_1', defined at:
  File "/root/venv2/bin/lumi", line 11, in <module>
    sys.exit(cli())
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/venv2/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/root/venv2/local/lib/python2.7/site-packages/luminoth/train.py", line 307, in train
    config, environment=environment
  File "/root/venv2/local/lib/python2.7/site-packages/luminoth/train.py", line 67, in run
    total_loss = model.loss(prediction_dict)
  File "/root/venv2/local/lib/python2.7/site-packages/luminoth/models/fasterrcnn/fasterrcnn.py", line 192, in loss
    prediction_dict['classification_prediction']
  File "/root/venv2/local/lib/python2.7/site-packages/luminoth/models/fasterrcnn/rcnn.py", line 391, in loss
    sigma=self._l1_sigma
  File "/root/venv2/local/lib/python2.7/site-packages/luminoth/utils/losses.py", line 22, in smooth_l1_loss
    diff = bbox_prediction - bbox_target
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 862, in binary_op_wrapper
    return func(x, y, name=name)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8318, in sub
    "Sub", x=x, y=y, name=name)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helpe
    op_def=op_def)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3272, in create_op
    op_def=op_def)
  File "/root/venv2/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1768, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [3,4] vs. [9,4]
         [[{{node losses/RCNNLoss/sub_1}} = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](losses/RCNNLos
atherV2)]]

(venv2) root@3e5d7b5d1a41:~#

[sbochkar]

Does the number of classes specified in the config match the number of classes in your dataset?

[keng-yu]

Yes, it matches. Or there will be different errors. Also tried COCO 2017 train dataset, the same error "InvalidArgumentError" happens too.

[kshitijagrwl]

I get a similar error on train,evaluate and predict as well on the COCO 2017 and COCO 2014 dataset. I have pytorch 0.4 installed.

r$ lumi train -c train.yml INFO:tensorflow:Training 279 vars from pretrained module; from "truncated_base_network/resnet_v1_101/block2/unit_1/bottleneck_v1/shortcut/weights:0" to "truncated_base_network/resnet_v1_101/block4/unit_3/bottleneck_v1/conv3/BatchNorm/beta:0". /home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " INFO:tensorflow:Starting training for <luminoth.models.fasterrcnn.fasterrcnn.FasterRCNN object at 0x7f0c69339b10> WARNING:tensorflow:From /home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py:118: initialize_local_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02. Instructions for updating: Use tf.local_variables_initializer instead. INFO:tensorflow:ImageVisHook was created with mode = "train" INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. 2018-12-04 04:23:59.919368: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA 2018-12-04 04:24:02.272545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62 pciBusID: 0000:81:00.0 totalMemory: 10.92GiB freeMemory: 10.76GiB 2018-12-04 04:24:02.647077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 1 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62 pciBusID: 0000:9e:00.0 totalMemory: 10.92GiB freeMemory: 10.76GiB 2018-12-04 04:24:02.649202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1 2018-12-04 04:24:03.581040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-12-04 04:24:03.581113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 1 2018-12-04 04:24:03.581133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N Y 2018-12-04 04:24:03.581143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1: Y N 2018-12-04 04:24:03.581729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10413 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:81:00.0, compute capability: 6.1) 2018-12-04 04:24:03.810598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10413 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:9e:00.0, compute capability: 6.1) INFO:tensorflow:Restoring parameters from /home/ml/.luminoth/resnet_v1_101.ckpt INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. Traceback (most recent call last): File "/home/ml/anaconda3/envs/plt-tf-py2/bin/lumi", line 11, in sys.exit(cli()) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 535, in invoke return callback(args, kwargs) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/luminoth/train.py", line 307, in train config, environment=environment File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/luminoth/train.py", line 239, in run ], options=run_options) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 567, in run run_metadata=run_metadata) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1043, in run run_metadata=run_metadata) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1134, in run raise six.reraise(original_exc_info) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1119, in run return self._sess.run(args, kwargs) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1191, in run run_metadata=run_metadata) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 971, in run return self._sess.run(*args, **kwargs) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 900, in run run_metadata_ptr) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run run_metadata) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [0,4] vs. [25,4] [[Node: losses/RCNNLoss/sub_1 = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](losses/RCNNLoss/bbox_offset_cleaned/GatherV2, losses/RCNNLoss/bbox_offsets_target_labeled/GatherV2)]] [[Node: fasterrcnn/rcnn/rcnn_proposal_1/GatherV2_16/_3769 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_11223_fasterrcnn/rcnn/rcnn_proposal_1/GatherV2_16", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op u'losses/RCNNLoss/sub_1', defined at: File "/home/ml/anaconda3/envs/plt-tf-py2/bin/lumi", line 11, in sys.exit(cli()) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/click/core.py", line 535, in invoke return callback(args, **kwargs) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/luminoth/train.py", line 307, in train config, environment=environment File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/luminoth/train.py", line 67, in run total_loss = model.loss(prediction_dict) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/luminoth/models/fasterrcnn/fasterrcnn.py", line 192, in loss prediction_dict['classification_prediction'] File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/luminoth/models/fasterrcnn/rcnn.py", line 391, in loss sigma=self._l1_sigma File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/luminoth/utils/losses.py", line 22, in smooth_l1_loss diff = bbox_prediction - bbox_target File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 979, in binary_op_wrapper return func(x, y, name=name) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8009, in sub "Sub", x=x, y=y, name=name) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op op_def=op_def) File "/home/ml/anaconda3/envs/plt-tf-py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1718, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [0,4] vs. [25,4] [[Node: losses/RCNNLoss/sub_1 = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](losses/RCNNLoss/bbox_offset_cleaned/GatherV2, losses/RCNNLoss/bbox_offsets_target_labeled/GatherV2)]] [[Node: fasterrcnn/rcnn/rcnn_proposal_1/GatherV2_16/_3769 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_11223_fasterrcnn/rcnn/rcnn_proposal_1/GatherV2_16", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

[meyerjo]

I think this is fixed with #261