search

GeorgeSeif / Semantic-Segmentation-Suite

Semantic Segmentation Suite in TensorFlow. Implement, train, and test new Semantic Segmentation models easily!
2.5k stars 880 forks source link

Unable to run training on Deeplab_V3+ using InceptionV4 frontend #149

Open bilel-bj opened 5 years ago

bilel-bj commented 5 years ago

After the running of this command: python train.py --num_epoch 1 --model DeepLabV3_plus --frontend InceptionV4

The training could not load successfully the pre-trained weights of InceptionV4, although it worked successfully with ResNet101, ResNet50, and ResNet152

***** Begin training *****
Dataset --> CamVid
Model --> DeepLabV3_plus
Crop Height --> 512
Crop Width --> 512
Num Epochs --> 1
Batch Size --> 1
Num Classes --> 32
Data Augmentation:
    Vertical Flip --> False
    Horizontal Flip --> False
    Brightness Alteration --> None
    Rotation --> None

Traceback (most recent call last):
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1039, in _do_call
    return fn(*args)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1021, in _run_fn
    status, run_metadata)
  File "/home/riotu/anaconda3/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: All dimensions except 3 must match. Input 1 has shape [1 128 128 256] and doesn't match input 0 with shape [1 126 126 48].
     [[Node: gradients/concat_1_grad/ConcatOffset = ConcatOffset[N=2, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/concat_1_grad/mod, gradients/concat_1_grad/ShapeN, gradients/concat_1_grad/ShapeN:1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_v2.py", line 202, in <module>
    _,current=sess.run([opt,loss],feed_dict={net_input:input_image_batch,net_output:output_image_batch})
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 778, in run
    run_metadata_ptr)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 982, in _run
    feed_dict_string, options, run_metadata)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1032, in _do_run
    target_list, options, run_metadata)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1052, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: All dimensions except 3 must match. Input 1 has shape [1 128 128 256] and doesn't match input 0 with shape [1 126 126 48].
     [[Node: gradients/concat_1_grad/ConcatOffset = ConcatOffset[N=2, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/concat_1_grad/mod, gradients/concat_1_grad/ShapeN, gradients/concat_1_grad/ShapeN:1)]]

Caused by op 'gradients/concat_1_grad/ConcatOffset', defined at:
  File "train_v2.py", line 104, in <module>
    opt = tf.train.RMSPropOptimizer(learning_rate=0.0001, decay=0.995).minimize(loss, var_list=[var for var in tf.trainable_variables()])
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 315, in minimize
    grad_loss=grad_loss)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 386, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 560, in gradients
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 368, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 560, in <lambda>
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/array_grad.py", line 194, in _ConcatGradV2
    op, grad, start_value_index=0, end_value_index=-1, dim_index=-1)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/array_grad.py", line 124, in _ConcatGradHelper
    offset = gen_array_ops._concat_offset(non_neg_concat_dim, sizes)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 493, in _concat_offset
    shape=shape, name=name)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

...which was originally created as op 'concat_1', defined at:
  File "train_v2.py", line 100, in <module>
    network, init_fn = model_builder.build_model(model_name=args.model, frontend=args.frontend, net_input=net_input, num_classes=num_classes, crop_width=args.crop_width, crop_height=args.crop_height, is_training=True)
  File "/home/riotu/Dropbox/Urban_areas_segmentation_from_uav_journal_paper/Code/builders/model_builder.py", line 80, in build_model
    network, init_fn = build_deeplabv3_plus(net_input, preset_model = model_name, frontend=frontend, num_classes=num_classes, is_training=is_training)
  File "/home/riotu/Dropbox/Urban_areas_segmentation_from_uav_journal_paper/Code/models/DeepLabV3_plus.py", line 88, in build_deeplabv3_plus
    net = tf.concat((encoder_features, decoder_features), axis=3)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 1034, in concat
    name=name)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 519, in _concat_v2
    name=name)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/riotu/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): All dimensions except 3 must match. Input 1 has shape [1 128 128 256] and doesn't match input 0 with shape [1 126 126 48].
     [[Node: gradients/concat_1_grad/ConcatOffset = ConcatOffset[N=2, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/concat_1_grad/mod, gradients/concat_1_grad/ShapeN, gradients/concat_1_grad/ShapeN:1)]]