IndexError when freezing imagenet resnet model

Spugnam commented 6 years ago

System information

What is the top-level directory of the model you are using: https://github.com/tensorflow/models/tree/master/official/resnet
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No - using official resnet model with imagenet data (built using models/research/inception/inception/data/build_imagenet_data.py)
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): ('v1.8.0-0-g93bc2e2072', '1.8.0')
Bazel version (if compiling from source): NA
CUDA/cuDNN version: CUDA Version 9.1.85 CUDNN 7.1.3
GPU model and memory: 2 * GeForce GTX 1080 Ti major: 6 minor: 1
Exact command to reproduce: python3 /home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py \ --input_graph /mnt/sda1/tensorflow_logs/imagenet_resnet/graph.pbtxt \ --input_checkpoint /mnt/sda1/tensorflow_logs/imagenet_resnet/model.ckpt-9335 \ --input_binary false \ --output_graph /mnt/sda1/tensorflow_logs/imagenet_resnet/frozen_resnet_v1.pb \ --output_node_names ArgMax \ --clear_devices

Describe the problem

The above command generates this stack: Traceback (most recent call last): File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 384, in run_main() File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 381, in run_main app.run(main=my_main, argv=[sys.argv[0]] + unparsed) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run _sys.exit(main(argv)) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 380, in my_main = lambda unused_args: main(unused_args, flags) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 274, in main flags.saved_model_tags, checkpoint_version) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 256, in freeze_graph checkpoint_version=checkpoint_version) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 130, in freeze_graph_with_def_protos var_list=var_list, write_version=checkpoint_version) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1342, in init self.build() File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1351, in build self._build(self._filename, build_save=True, build_restore=True) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1388, in _build build_save=build_save, build_restore=build_restore) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 817, in _build_internal saveables = self._ValidateAndSliceInputs(names_to_saveables) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 724, in _ValidateAndSliceInputs variable, "", name) File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 198, in init self.handle_op = var.op.inputs[0] File "/home/spugnam/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2103, in getitem return self._inputs[i] IndexError: list index out of range

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

In an effort to retrieve the correct output_node_names I ran

for op in tf.get_default_graph().get_operations():                           
    print(op.name)

and the ouput was:

[...] resnet_model/Mean resnet_model/final_reduce_mean resnet_model/Reshape/shape resnet_model/Reshape resnet_model/dense/kernel/Initializer/random_uniform/shape resnet_model/dense/kernel/Initializer/random_uniform/min resnet_model/dense/kernel/Initializer/random_uniform/max resnet_model/dense/kernel/Initializer/random_uniform/RandomUniform resnet_model/dense/kernel/Initializer/random_uniform/sub resnet_model/dense/kernel/Initializer/random_uniform/mul resnet_model/dense/kernel/Initializer/random_uniform resnet_model/dense/kernel resnet_model/dense/kernel/Assign resnet_model/dense/kernel/read resnet_model/dense/bias/Initializer/zeros resnet_model/dense/bias resnet_model/dense/bias/Assign resnet_model/dense/bias/read resnet_model/dense/MatMul resnet_model/dense/BiasAdd resnet_model/final_dense ArgMax/dimension ArgMax softmax_tensor softmax_cross_entropy_loss/labels_stop_gradient softmax_cross_entropy_loss/xentropy/Rank softmax_cross_entropy_loss/xentropy/Shape softmax_cross_entropy_loss/xentropy/Rank_1 softmax_cross_entropy_loss/xentropy/Shape_1 softmax_cross_entropy_loss/xentropy/Sub/y softmax_cross_entropy_loss/xentropy/Sub softmax_cross_entropy_loss/xentropy/Slice/begin softmax_cross_entropy_loss/xentropy/Slice/size softmax_cross_entropy_loss/xentropy/Slice softmax_cross_entropy_loss/xentropy/concat/values_0 softmax_cross_entropy_loss/xentropy/concat/axis softmax_cross_entropy_loss/xentropy/concat softmax_cross_entropy_loss/xentropy/Reshape softmax_cross_entropy_loss/xentropy/Rank_2 softmax_cross_entropy_loss/xentropy/Shape_2 softmax_cross_entropy_loss/xentropy/Sub_1/y softmax_cross_entropy_loss/xentropy/Sub_1 softmax_cross_entropy_loss/xentropy/Slice_1/begin softmax_cross_entropy_loss/xentropy/Slice_1/size softmax_cross_entropy_loss/xentropy/Slice_1 softmax_cross_entropy_loss/xentropy/concat_1/values_0 softmax_cross_entropy_loss/xentropy/concat_1/axis softmax_cross_entropy_loss/xentropy/concat_1 softmax_cross_entropy_loss/xentropy/Reshape_1 softmax_cross_entropy_loss/xentropy softmax_cross_entropy_loss/xentropy/Sub_2/y softmax_cross_entropy_loss/xentropy/Sub_2 softmax_cross_entropy_loss/xentropy/Slice_2/begin softmax_cross_entropy_loss/xentropy/Slice_2/size softmax_cross_entropy_loss/xentropy/Slice_2 softmax_cross_entropy_loss/xentropy/Reshape_2 softmax_cross_entropy_loss/assert_broadcastable/weights softmax_cross_entropy_loss/assert_broadcastable/weights/shape softmax_cross_entropy_loss/assert_broadcastable/weights/rank softmax_cross_entropy_loss/assert_broadcastable/values/shape softmax_cross_entropy_loss/assert_broadcastable/values/rank softmax_cross_entropy_loss/assert_broadcastable/static_scalar_check_success softmax_cross_entropy_loss/ToFloat_1/x softmax_cross_entropy_loss/Mul softmax_cross_entropy_loss/Const softmax_cross_entropy_loss/Sum softmax_cross_entropy_loss/num_present/Equal/y softmax_cross_entropy_loss/num_present/Equal softmax_cross_entropy_loss/num_present/zeros_like softmax_cross_entropy_loss/num_present/ones_like/Shape softmax_cross_entropy_loss/num_present/ones_like/Const softmax_cross_entropy_loss/num_present/ones_like softmax_cross_entropy_loss/num_present/Select softmax_cross_entropy_loss/num_present/broadcast_weights/assert_broadcastable/weights/shape softmax_cross_entropy_loss/num_present/broadcast_weights/assert_broadcastable/weights/rank softmax_cross_entropy_loss/num_present/broadcast_weights/assert_broadcastable/values/shape softmax_cross_entropy_loss/num_present/broadcast_weights/assert_broadcastable/values/rank softmax_cross_entropy_loss/num_present/broadcast_weights/assert_broadcastable/static_scalar_check_success softmax_cross_entropy_loss/num_present/broadcast_weights/ones_like/Shape softmax_cross_entropy_loss/num_present/broadcast_weights/ones_like/Const softmax_cross_entropy_loss/num_present/broadcast_weights/ones_like softmax_cross_entropy_loss/num_present/broadcast_weights softmax_cross_entropy_loss/num_present/Const softmax_cross_entropy_loss/num_present softmax_cross_entropy_loss/Const_1 softmax_cross_entropy_loss/Sum_1 softmax_cross_entropy_loss/Greater/y softmax_cross_entropy_loss/Greater softmax_cross_entropy_loss/Equal/y softmax_cross_entropy_loss/Equal softmax_cross_entropy_loss/ones_like/Shape softmax_cross_entropy_loss/ones_like/Const softmax_cross_entropy_loss/ones_like softmax_cross_entropy_loss/Select softmax_cross_entropy_loss/div softmax_cross_entropy_loss/zeros_like softmax_cross_entropy_loss/value cross_entropy cross_entropy_1/tags cross_entropy_1 L2Loss L2Loss_1 L2Loss_2 L2Loss_3 L2Loss_4 L2Loss_5 L2Loss_6 L2Loss_7 L2Loss_8 L2Loss_9 L2Loss_10 L2Loss_11 L2Loss_12 L2Loss_13 L2Loss_14 L2Loss_15 L2Loss_16 L2Loss_17 L2Loss_18 L2Loss_19 L2Loss_20 L2Loss_21 L2Loss_22 AddN mul/x mul l2_loss/tags l2_loss add ArgMax_1/dimension ArgMax_1 Equal ToFloat accuracy/total/Initializer/zeros accuracy/total accuracy/total/Assign accuracy/total/read accuracy/count/Initializer/zeros accuracy/count accuracy/count/Assign accuracy/count/read accuracy/Size accuracy/ToFloat_1 accuracy/Const accuracy/Sum accuracy/AssignAdd accuracy/AssignAdd_1 accuracy/truediv accuracy/zeros_like accuracy/Greater accuracy/value accuracy/truediv_1 accuracy/zeros_like_1 accuracy/Greater_1 accuracy/update_op train_accuracy train_accuracy_1/tags train_accuracy_1

Note: for other models (e.g. tensorflow for poets 2) the output_node_names was accuracy/correct_prediction/Equal but nothing with 'prediction' in the resnet model (Note: the error actually happens before output_node_names is used).

Any hint at what the problem could be would be greatly appreciated.

yhliang2018 commented 6 years ago

Not sure if it's the issue of the way you use imagenet dataset, as there is a different imagenet data preprocessing in official model. Could you replace the research/inception build_imagenet_data.py with the official imagenet_preprocessing.py? Try it again and to see if the issue is gone.

Spugnam commented 6 years ago

I am absolutely using the official imagenet_preprocessing.py file, which is called in models/official/resnet/imagenet_main.py. The build_imagenet_data.py comes earlier in the process to download the images and their labels. I only followed the instructions of the official resnet model:

ImageNet

Setup

To begin, you will need to download the ImageNet dataset and convert it to TFRecord format. Follow along with the Inception guide in order to prepare the dataset.

yhliang2018 commented 6 years ago

@robieta Do you have any idea on this issue?

tung7970 commented 6 years ago

Try to export saved model, and use --input_saved_model_dir instead of --input_graph as input to freeze_graph. I had the same issue with official mnist and cifar10 model before I tried the saved model format. I figured the saved model has everything set up correctly for freeze_graph to work.

e.g. freeze_graph \ --input_saved_model_dir=cifar10_saved_model_dir \ --output_graph=cifar10.pb \ --output_node_names=ArgMax \ --clear_devices

Spugnam commented 6 years ago

Converted 107 variables to const ops. Tears of joy are flowing down my cheeks thanks!

alexcdot commented 6 years ago

This was the solution for me too on tensorflow 1.10.0

wt-huang commented 5 years ago

Closing as this is resolved

jiayiliu commented 5 years ago

Closing as this is resolved

This is only a work-around, the problem is still there. See #22029, #5387, and #24591

tensorflow / models