juanmanuelrq
commented
5 years ago During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 285, in
[[{{node gradients/DynamicPartition_grad/range/_7773}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_35384_gradients/DynamicPartition_grad/range", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'resnet_v2_101/block3/unit_1/bottleneck_v2/conv3/Conv2D', defined at:
File "train.py", line 285, in
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[10,1024,33,33] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node resnet_v2_101/block3/unit_1/bottleneck_v2/conv3/Conv2D}} = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](resnet_v2_101/block3/unit_1/bottleneck_v2/conv2/Relu, resnet_v2_101/block3/unit_1/bottleneck_v2/conv3/weights/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node gradients/DynamicPartition_grad/range/_7773}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_35384_gradients/DynamicPartition_grad/range", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
northeastsquare
prativadas
Curry-Christopher
The following problems occur when I run the train.py file.
File "train.py", line 285, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "train.py", line 267, in main
hooks=train_hooks,
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 354, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 1207, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 1237, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 1195, in _call_model_fn
model_fn_results = self._model_fn(features=features, kwargs)
File "/home/algolab/HDD/LHHS/DeepLab_v3_plus/tensorflow-deeplab-v3-plus-master/deeplab_model.py", line 172, in deeplabv3_plus_model_fn
logits = network(features, mode == tf.estimator.ModeKeys.TRAIN)
File "/home/algolab/HDD/LHHS/DeepLab_v3_plus/tensorflow-deeplab-v3-plus-master/deeplab_model.py", line 129, in model
{v.name.split(':')[0]: v for v in variables_to_restore})
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/training/checkpoint_utils.py", line 187, in init_from_checkpoint
_init_from_checkpoint, ckpt_dir_or_file, assignment_map)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/training/distribute.py", line 1053, in merge_call
return self._merge_call(merge_fn, *args, *kwargs)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/training/distribute.py", line 1061, in _merge_call
return merge_fn(self._distribution_strategy, args, kwargs)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/training/checkpoint_utils.py", line 194, in _init_from_checkpoint
reader = load_checkpoint(ckpt_dir_or_file)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/training/checkpoint_utils.py", line 64, in load_checkpoint
return pywrap_tensorflow.NewCheckpointReader(filename)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 326, in NewCheckpointReader
return CheckpointReader(compat.as_bytes(filepattern), status)
File "/home/algolab/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for PRE_TRAINED_MODEL
I have resnet_v2_101.ckpt in the following path. Why does this happen?
parser.add_argument('--pre_trained_model', type=str, default='./ini_checkpoints/resnet_v2_101/resnet_v2_101.ckpt', help='Path to the pre-trained model checkpoint.')