[variables_helper.py:141]Variable [SecondStageBoxPredictor/BoxEncodingPredictor/biases] is available in checkpoint, but has an incompatible shape with model.

gfjiyue commented 5 years ago

System information

What is the top-level directory of the model you are using: Object detection Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes, but I just edited the model_main.py at line 62:

config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir,  
 save_checkpoints_steps=500,
 tf_random_seed=1230, 
 keep_checkpoint_max=100)

OS Platform and Distribution: Ubuntu "16.04.2 LTS (Xenial Xerus)" TensorFlow installed from (source or binary): installed with Python's pip package TensorFlow version (use command below): ('v1.9.0-0-g25c197e023', '1.9.0') Bazel version: N/A CUDA/cuDNN version: CUDA 8.0 GPU model and memory: Tesla P40 22919MiB Exact command to reproduce: export CUDA_VISIBLE_DEVICES=3 python object_detection/model_main_my.py \ --pipeline_config_path=${train_path}/data/faster_rcnn_resnet101_t.config \ --model_dir=${train_path}/model \ --num_train_steps=50000 \ --sample_1_of_n_eval_examples=1 \ --alsologtostderr

Describe the problem

There always show warnings in the secondstage. I want to know if this in relation with the pretrain model and if it will influence the box detection precision.

Warnings:

W1209 21:57:14.465106 140428642035456 variables_helper.py:141] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[360]], model variable shape: [[4]]. This variable will not be initialized from the checkpoint. W1209 21:57:14.465503 140428642035456 variables_helper.py:141] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[2048, 360]], model variable shape: [[2048, 4]]. This variable will not be initialized from the checkpoint. W1209 21:57:14.465818 140428642035456 variables_helper.py:141] Variable [SecondStageBoxPredictor/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[91]], model variable shape: [[2]]. This variable will not be initialized from the checkpoint. W1209 21:57:14.466134 140428642035456 variables_helper.py:141] Variable [SecondStageBoxPredictor/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[2048, 91]], model variable shape: [[2048, 2]]. This variable will not be initialized from the checkpoint.

tensorflowbutler commented 5 years ago

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks. Bazel version

gfjiyue commented 5 years ago

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks. Bazel version Already updated.

GPhilo commented 4 years ago

It seems you changed the number of classes your model predicts from 90 to 1 (plus background), so the box predictor and the classifier heads can't load the pretrained weights (because they're specific to the original classes). This is expected (these variables are -mostly- what you train during fine-tuning). Will it affect box detection? If you don't fine-tune the model, yes, because the new variables are initialized with their default initializer's value (e.g., randomly initialized). With fine-tuning thought you'll be fine.

As a side note, I'm assuming that the class you're trying to predict is not one of the original 90. If that's not the case (and you don't want to fine-tune), either you use the original model and discard all results you're not interested in (but this wastes GPU memory & time because you make computations you don't need), or you manually slice the weights to keep only the ones relevant to you and load them manually in the model (a bit tricky, but not very difficult to do).

CoderGenJ commented 4 years ago

It seems you changed the number of classes your model predicts from 90 to 1 (plus background), so the box predictor and the classifier heads can't load the pretrained weights (because they're specific to the original classes). This is expected (these variables are -mostly- what you train during fine-tuning). Will it affect box detection? If you don't fine-tune the model, yes, because the new variables are initialized with their default initializer's value (e.g., randomly initialized). With fine-tuning thought you'll be fine.

As a side note, I'm assuming that the class you're trying to predict is not one of the original 90. If that's not the case (and you don't want to fine-tune), either you use the original model and discard all results you're not interested in (but this wastes GPU memory & time because you make computations you don't need), or you manually slice the weights to keep only the ones relevant to you and load them manually in the model (a bit tricky, but not very difficult to do).

I have a similar problem. Checkpoint shape: [[3, 3, 256, 24]], model variable shape: [[1, 1, 256, 24]]. This variable will not be initialized from the checkpoint. I don't understand that if the reason is class number changed, but why the kernel size is diffierent which is from [3,3] to [1,1].?

tensorflow / models