search

NVIDIA-AI-IOT / tf_to_trt_image_classification

Image classification with NVIDIA TensorRT from TensorFlow models.
BSD 3-Clause "New" or "Revised" License
454 stars 155 forks source link

Converting to frozen graph failed on Jetson TX2 #4

Open Chomolungma opened 6 years ago

Chomolungma commented 6 years ago

When I run the scripts/models_to_frozen_graphs.py script, I got below error ion Jetson TX2. Any comment: ... 018-03-13 18:52:01.449359: I tensorflow/core/common_runtime/bfc_allocator.cc:684] Sum Total of in-use chunks: 152.63MiB 2018-03-13 18:52:01.449608: I tensorflow/core/common_runtime/bfc_allocator.cc:686] Stats: Limit: 374509568 InUse: 160042752 MaxInUse: 271571968 NumAllocs: 10753 MaxAllocSize: 107122688

2018-03-13 18:52:01.449952: W tensorflow/core/common_runtime/bfc_allocator.cc:277] __*x***xx**__*****__ Traceback (most recent call last): File "scripts/models_to_frozen_graphs.py", line 63, in sess=tf_sess File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1750, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1128, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1344, in _do_run options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1363, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized. [[Node: save/RestoreV2_27/_11 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_78_save/RestoreV2_27", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

ghost commented 6 years ago

We have encountered out-of-memory before on the TX2 when running the batch conversion script (scripts/models_to_frozen_graphs.py) immediately after performing the other installation steps.

Could you try restarting the Jetson TX2 and report back the results of running the above script?

lesonkorenac commented 6 years ago

I had the same problem and it disappeared after restarting Jetson TX2 as you suggested.