search

tensorflow / models

Models and examples built with TensorFlow
Other
77.01k stars 45.78k forks source link

Possible mismatch between the provided ResNet V1 50 pre-trained ckpt and pb #7975

Closed tigert1998 closed 4 years ago

tigert1998 commented 4 years ago

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

Also, please understand that many of the models included in this repository are experimental and research-style code. If you open a GitHub issue, here is our policy:

  1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
  2. The form below must be filled out.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

I found that maybe there's a mismatch between the provided ResNet V1 50 pre-trained model ckpt and generated resnet_v1_50.pb.

There's how to reproduce:

python .\models\research\slim\export_inference_graph.py --model_name=resnet_v1_50 --output_file=./resnet_v1_50_var.pb
# download ckpt from http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz and extract to the working directory
python ~\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\tools\freeze_graph.py --input_graph=.\resnet_v1_50_var.pb --input_checkpoint=.\resnet_v1_50.ckpt --input_binary=true --output_graph=.\resnet_v1_50.pb --output_node_names=resnet_v1_50/predictions/Reshape_1

Source code / logs

WARNING:tensorflow:From C:\\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py:127: checkpoint_exists (from 
tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W1221 17:41:38.434148 19332 deprecation.py:323] From C:\\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-21 17:41:39.003578: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  AVX AVX2
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2019-12-21 17:41:39.022982: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 12. Tune using inter_op_parallelism_threads for best performance.
INFO:tensorflow:Restoring parameters from .\resnet_v1_50.ckpt
I1221 17:41:39.283333 19332 saver.py:1284] Restoring parameters from .\resnet_v1_50.ckpt
Traceback (most recent call last):
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
    return fn(*args)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
         [[{{node save/Assign_265}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 1290, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
    run_metadata_ptr)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
    run_metadata)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
         [[node save/Assign_265 (defined at Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]

Original stack trace for 'save/Assign_265':
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 491, in <module>
    run_main()
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 487, in run_main
    app.run(main=my_main, argv=[sys.argv[0]] + unparsed)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 486, in <lambda>
    my_main = lambda unused_args: main(unused_args, flags)
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 378, in main
    flags.saved_model_tags, checkpoint_version)
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 361, in freeze_graph
    checkpoint_version=checkpoint_version)
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 190, in freeze_graph_with_def_protos     
    var_list=var_list, write_version=checkpoint_version)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 828, in __init__
    self.build()
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 878, in _build
    build_restore=build_restore)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saving\saveable_object_util.py", line 73, in restore
    self.op.get_shape().is_fully_defined())
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\ops\state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\ops\gen_state_ops.py", line 65, in assign
    use_locking=use_locking, name=name)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 491, in <module>
    run_main()
  File "C:\\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 487, in run_main
    app.run(main=my_main, argv=[sys.argv[0]] + unparsed)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "C:\\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 486, in <lambda>
    my_main = lambda unused_args: main(unused_args, flags)
  File "C:\\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 378, in main
    flags.saved_model_tags, checkpoint_version)
  File "C:\\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 361, in freeze_graph
    checkpoint_version=checkpoint_version)
  File "C:\\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 209, in freeze_graph_with_def_protos  
    saver.restore(sess, input_checkpoint)
  File "C:\Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 1326, in restore
    err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
         [[node save/Assign_265 (defined at Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]

Original stack trace for 'save/Assign_265':
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 491, in <module>
    run_main()
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 487, in run_main
    app.run(main=my_main, argv=[sys.argv[0]] + unparsed)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 486, in <lambda>
    my_main = lambda unused_args: main(unused_args, flags)
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 378, in main
    flags.saved_model_tags, checkpoint_version)
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 361, in freeze_graph
    checkpoint_version=checkpoint_version)
  File "\Users\\v-xiat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\tensorflow_core\\python\\tools\\freeze_graph.py", line 190, in freeze_graph_with_def_protos     
    var_list=var_list, write_version=checkpoint_version)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 828, in __init__
    self.build()
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 878, in _build
    build_restore=build_restore)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\training\saving\saveable_object_util.py", line 73, in restore
    self.op.get_shape().is_fully_defined())
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\ops\state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\ops\gen_state_ops.py", line 65, in assign
    use_locking=use_locking, name=name)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "Users\v-xiat\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()
sguada commented 4 years ago

Make sure to use 1001 classes there is one extra for background.