error coming while giving while giving the path of pretrained model

[ravi7248]

I am trying to run your project everything is working fine but I am confused about which path I have to give in --initial-checkpoint. I tried giving the path of pre-trained inception V4, resnet_v1_50 and inception-resnet-v2 but I am getting the error at first epoch of the model and I also commented the line no 8 from evaluation.evaluation_helper import get_evaluation_summary_writer and lines related to this in trainer_preid.py but still getting the error. I highlighted the error in the last as well. Can anyone suggest the path of which we have to give in the parameter --initial-checkpoint

Running training of epoch 1 of 100:

Read 12936 image paths for processing for dataset_part: train WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:From /content/pose-sensitive-embedding/helper/model_helper.py:14: slice_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensor_slices(tuple(tensor_list)).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs). If shuffle=False, omit the .shuffle(...). WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:373: range_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.range(limit).shuffle(limit).repeat(num_epochs). If shuffle=False, omit the .shuffle(...). WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:319: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs). If shuffle=False, omit the .shuffle(...). WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:189: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensors(tensor).repeat(num_epochs). WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:112: RefVariable.count_up_to (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Prefer Dataset.range instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py:2522: count_up_to (from tensorflow.python.ops.state_ops) is deprecated and will be removed in a future version. Instructions for updating: Prefer Dataset.range instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:198: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:198: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From /content/pose-sensitive-embedding/datasets/Dataset.py:52: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

WARNING:tensorflow:From /content/pose-sensitive-embedding/datasets/Dataset.py:56: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From /content/pose-sensitive-embedding/datasets/Dataset.py:85: The name tf.random_crop is deprecated. Please use tf.image.random_crop instead.

WARNING:tensorflow:From /content/pose-sensitive-embedding/helper/model_helper.py:21: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.batch(batch_size) (or padded_batch(...) if dynamic_pad=True). WARNING:tensorflow:From /content/pose-sensitive-embedding/nets/resnet_utils.py:237: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From /content/pose-sensitive-embedding/nets/resnet_v1.py:181: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. Initial run: Loading initial_checkpoint: /content/model.ckpt-81450.data-00000-of-00001 WARNING:tensorflow:From /content/pose-sensitive-embedding/helper/checkpoint_helper.py:27: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

2019-11-12 11:56:07.040967: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open /content/model.ckpt-81450.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator? Traceback (most recent call last): File "/content/pose-sensitive-embedding/trainer_preid.py", line 92, in main() File "/content/pose-sensitive-embedding/trainer_preid.py", line 86, in main args.learning_rate_decay_rate, not args.no_evaluation, args.learning_rate_decay_steps) File "/content/pose-sensitive-embedding/trainer_preid.py", line 38, in start_training run_training(dataset=dataset, batch_size=batch_size, batch_threads=batch_threads, epoch=epoch, estimator=estimator, num_epochs=num_epochs, image_size=image_size) File "/content/pose-sensitive-embedding/trainer_preid.py", line 50, in run_training estimator.train(input_fn=train_input_function) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1191, in _train_model_default features, labels, ModeKeys.TRAIN, self.config) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn model_fn_results = self._model_fn(features=features, kwargs) File "/content/pose-sensitive-embedding/helper/model_helper.py", line 49, in model_fn check_init_from_initial_checkpoint(output_directory, initial_checkpoint, checkpoint_exclude_scopes, ignore_missing_variables) File "/content/pose-sensitive-embedding/helper/checkpoint_helper.py", line 14, in check_init_from_initial_checkpoint init_from_checkpoint(initial_checkpoint, checkpoint_exclude_scopes, ignore_missing_variables) File "/content/pose-sensitive-embedding/helper/checkpoint_helper.py", line 20, in init_from_checkpoint tf.contrib.framework.init_from_checkpoint(initial_checkpoint, variables_dictionary) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/framework/python/framework/checkpoint_utils.py", line 222, in init_from_checkpoint reader = load_checkpoint(checkpoint_dir) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/framework/python/framework/checkpoint_utils.py", line 64, in load_checkpoint return train.NewCheckpointReader(filename) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/pywrap_tensorflow_internal.py", line 873, in NewCheckpointReader return CheckpointReader(compat.as_bytes(filepattern)) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/pywrap_tensorflow_internal.py", line 885, in init this = _pywrap_tensorflow_internal.new_CheckpointReader(filename) tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file /content/model.ckpt-81450.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?**