No Variables to restore？

1453042287 commented 5 years ago

Information

Please specify the following information when submitting an issue:

What are your command line arguments?: python train.py --model="DeepLabV3_plus" --frontend="MobileNetV2"
Have you written any custom code?: no
What have you done to try and solve this issue?: nothing
TensorFlow version?: 1.11
Describe the problem

after download the mobilenetv2 pre-trained model, the log of tensorflow still tell me there is no variables to restore!

Source code / logs

Preparing the model ... WARNING:tensorflow:Variable mobilenet_v2/Conv/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/depthwise/depthwise_weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/depthwise/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/depthwise/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/depthwise/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/depthwise/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/project/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/project/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/project/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/project/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv/project/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/expand/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/expand/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/expand/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/expand/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/expand/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/depthwise/depthwise_weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/depthwise/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/depthwise/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/depthwise/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/depthwise/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/project/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/project/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/project/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/project/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_1/project/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/expand/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/expand/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/expand/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/expand/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/expand/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/depthwise/depthwise_weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/depthwise/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/depthwise/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/depthwise/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/depthwise/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/project/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/project/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/project/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/project/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_2/project/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt ... ... ... WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/expand/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/expand/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/expand/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/expand/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/expand/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/depthwise/depthwise_weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/depthwise/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/depthwise/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/depthwise/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/depthwise/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/project/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/project/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/project/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/project/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/expanded_conv_16/project/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv_1/weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv_1/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv_1/BatchNorm/beta missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv_1/BatchNorm/moving_mean missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable mobilenet_v2/Conv_1/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:No Variables to restore

Sixkplus commented 5 years ago

When I use BiSeNet based on mobilenet2, i faced the same warning information. Infact, the standard checkpoint of mobilenet_v2 uses scope name 'MobileNetV2', but not the authors' implementation 'mobilenet_v2' in 'frontend_builder.py'. When I modified this issue, my program doesn't warn me anymore. Have a try, :D

1453042287 commented 5 years ago

@Sixkplus thanks a lot！

GeorgeSeif commented 5 years ago

Hi there @Sixkplus

Would you mind sharing the piece of code you modified or submitting a pull request?

Thanks!

Sixkplus commented 5 years ago

In frontend_builder.py, modify the code related to mobilenet as follows.

elif frontend == 'MobileNetV2': with slim.arg_scope(mobilenet_v2.training_scope()): logits, end_points = mobilenet_v2.mobilenet(inputs, is_training=is_training, scope='MobilenetV2', base_only=True) frontend_scope='mobilenet_v2' init_fn = slim.assign_from_checkpoint_fn(model_path=os.path.join(pretrained_dir, 'mobilenet_v2.ckpt'), var_list=slim.get_model_variables('MobilenetV2'), ignore_missing_vars=True)

In fact, I also has another related problem. When I perform testing, the model won't work well if I set 'is_Training' to be False, i.e. it couldn't load the checkpoint as expected, but it works well if I set this parameter to be True.

zhangyunming commented 5 years ago

@Sixkplus hi，when i train the bisenet with Mobilenetv2，i modify the code as you said in frontend_builder.py, but the same warning occured, as followed:

WARNING:tensorflow:Variable MobileNetV2/expanded_conv_13/depthwise/BatchNorm/gamma missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable MobileNetV2/expanded_conv_16/depthwise/depthwise_weights missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:Variable MobileNetV2/expanded_conv_1/project/BatchNorm/moving_variance missing in checkpoint models/mobilenet_v2.ckpt WARNING:tensorflow:No Variables to restore WARNING:tensorflow:From /disk3t-2/zym/Semantic-Segmentation-Suite-master/models/BiSeNet.py:33: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead Traceback (most recent call last): File "train.py", line 95, in network, init_fn = model_builder.build_model(model_name=args.model, frontend=args.frontend, net_input=net_input, num_classes=num_classes, crop_width=args.crop_width, crop_height=args.crop_height, is_training=True) File "/disk3t-2/zym/Semantic-Segmentation-Suite-master/builders/model_builder.py", line 89, in build_model network, init_fn = build_bisenet(net_input, preset_model = model_name, frontend=frontend, num_classes=num_classes, is_training=is_training) File "/disk3t-2/zym/Semantic-Segmentation-Suite-master/models/BiSeNet.py", line 87, in build_bisenet net_4 = AttentionRefinementModule(end_points['pool4'], n_filters=512) File "/disk3t-2/zym/Semantic-Segmentation-Suite-master/models/BiSeNet.py", line 39, in AttentionRefinementModule net = tf.multiply(inputs, net) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 331, in multiply return gen_math_ops.mul(x, y, name) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 4689, in mul "Mul", x=x, y=y, name=name) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3292, in create_op compute_device=compute_device) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3332, in _create_op_helper set_shapes_for_outputs(op) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2496, in set_shapes_for_outputs return _set_shapes_for_outputs(op) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2469, in _set_shapes_for_outputs shapes = shape_func(op) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2399, in call_with_requiring return call_cpp_shape_fn(op, require_shape_fn=True) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 627, in call_cpp_shape_fn require_shape_fn) File "/home/os/.local/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 691, in _call_cpp_shape_fn_impl raise ValueError(err.message) ValueError: Dimensions must be equal, but are 64 and 512 for 'Mul' (op: 'Mul') with input shapes: [?,?,?,64], [?,1,1,512]. os@os-l3:/disk3t-2/zym/Semantic-Segmentation-Suite-master$

another question: if i trained mobile2+bisenet, how can i do not use the pretrained model and train from Scratch

1453042287 commented 5 years ago

@zhangyunming No Variables to restore = from scratch and maybe you need to check the name scope in your checkpoint file，where did you download the mobilenet_v2.ckpt？i just get it from the https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet

GeorgeSeif / Semantic-Segmentation-Suite