search

jeng1220 / KerasToTensorRT

This is a simple demonstration for running Keras model model on Tensorflow with TensorRT integration(TFTRT) or on TensorRT directly without invoking "freeze_graph.py".
67 stars 23 forks source link

error about class TftrtEngine #1

Closed Airyzf closed 5 years ago

Airyzf commented 5 years ago 
  def infer(self, x):
    num_tests = x.shape[0]
    y = np.empty((num_tests, self.y_tensor.shape[1]), np.float32)
    batch_size = self.batch_size

    for i in range(0, num_tests, batch_size):
      x_part = x[i : i + batch_size]
      y_part = self.sess.run(self.y_tensor,
        feed_dict={self.x_tensor: x_part})
      y[i : i + batch_size] = y_part
    return y

Hi, I try to run tftrt_example.py and have some confuses about the class EftrtEngine. There are tow attributes named self.y_tensor and self.x_tensor have no defined. Please help.

jeng1220 commented 5 years ago

class TftrtEngine inherits class TfEngine. see line 39, 40.

Airyzf commented 5 years ago

class TftrtEngine inherits class TfEngine. see line 39, 40.

oh i see ,thanks for your help, i just want to run TftrtEngine so annotate the line 112-117 , but i got an other exception now, i run in the tx2 ,what is wrong with me.

2018-10-19 17:52:58.577855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1485] Adding visible gpu devices: 0 2018-10-19 17:52:58.578004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:966] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-10-19 17:52:58.578036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] 0 2018-10-19 17:52:58.578064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:985] 0: N 2018-10-19 17:52:58.578177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1098] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3527 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2) 2018-10-19 17:53:02.416762: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.59GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2018-10-19 17:53:03.144163: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.17GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2018-10-19 17:53:13.156435: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.37GiB. Current allocation summary follows. 2018-10-19 17:53:13.156543: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (256): Total Chunks: 3, Chunks in use: 3. 768B allocated for chunks. 768B in use in bin. 424B client-requested in use in bin. 2018-10-19 17:53:13.156601: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (512): Total Chunks: 1, Chunks in use: 1. 512B allocated for chunks. 512B in use in bin. 512B client-requested in use in bin. 2018-10-19 17:53:13.156641: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (1024): Total Chunks: 2, Chunks in use: 2. 2.5KiB allocated for chunks. 2.5KiB in use in bin. 2.1KiB client-requested in use in bin. 2018-10-19 17:53:13.156673: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.156707: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (4096): Total Chunks: 1, Chunks in use: 1. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2018-10-19 17:53:13.156737: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.156767: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.156828: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.156868: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (65536): Total Chunks: 2, Chunks in use: 2. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin. 2018-10-19 17:53:13.156900: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.156930: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.156959: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.156990: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.157042: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.157078: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (4194304): Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin. 2018-10-19 17:53:13.157109: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.157139: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.157169: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.157199: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.157296: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-10-19 17:53:13.157338: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (268435456): Total Chunks: 4, Chunks in use: 2. 3.44GiB allocated for chunks. 2.18GiB in use in bin. 2.18GiB client-requested in use in bin. 2018-10-19 17:53:13.157376: I tensorflow/core/common_runtime/bfc_allocator.cc:626] Bin for 1.37GiB was 256.00MiB, Chunk State: 2018-10-19 17:53:13.157450: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Size: 436.78MiB | Requested Size: 1.1KiB | in_use: 0, prev: Size: 72.0KiB | Requested Size: 72.0KiB | in_use: 1 2018-10-19 17:53:13.157493: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Size: 855.10MiB | Requested Size: 29.91MiB | in_use: 0, prev: Size: 256B | Requested Size: 40B | in_use: 1, next: Size: 825.20MiB | Requested Size: 825.20MiB | in_use: 1 2018-10-19 17:53:13.157536: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc0e90000 of size 1280 2018-10-19 17:53:13.157566: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc0e90500 of size 1280 2018-10-19 17:53:13.157594: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc0e90a00 of size 256 2018-10-19 17:53:13.157620: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc0e90b00 of size 73728 2018-10-19 17:53:13.157645: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc0ea2b00 of size 256 2018-10-19 17:53:13.157671: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc0ea2c00 of size 4718592 2018-10-19 17:53:13.157696: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc1322c00 of size 512 2018-10-19 17:53:13.157721: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc1322e00 of size 5120 2018-10-19 17:53:13.157745: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xfc1324200 of size 256 2018-10-19 17:53:13.157770: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Free at 0xfc1324300 of size 896641280 2018-10-19 17:53:13.157802: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0xff6a3ec00 of size 865280000 2018-10-19 17:53:13.157828: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0x102a370c00 of size 1474560000 2018-10-19 17:53:13.157854: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Chunk at 0x10821b0c00 of size 73728 2018-10-19 17:53:13.157905: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Free at 0x10821c2c00 of size 457995264 2018-10-19 17:53:13.157930: I tensorflow/core/common_runtime/bfc_allocator.cc:651] Summary of in-use Chunks by size: 2018-10-19 17:53:13.157960: I tensorflow/core/common_runtime/bfc_allocator.cc:654] 3 Chunks of size 256 totalling 768B 2018-10-19 17:53:13.157987: I tensorflow/core/common_runtime/bfc_allocator.cc:654] 1 Chunks of size 512 totalling 512B 2018-10-19 17:53:13.158016: I tensorflow/core/common_runtime/bfc_allocator.cc:654] 2 Chunks of size 1280 totalling 2.5KiB 2018-10-19 17:53:13.158044: I tensorflow/core/common_runtime/bfc_allocator.cc:654] 1 Chunks of size 5120 totalling 5.0KiB 2018-10-19 17:53:13.158073: I tensorflow/core/common_runtime/bfc_allocator.cc:654] 2 Chunks of size 73728 totalling 144.0KiB 2018-10-19 17:53:13.158101: I tensorflow/core/common_runtime/bfc_allocator.cc:654] 1 Chunks of size 4718592 totalling 4.50MiB 2018-10-19 17:53:13.158131: I tensorflow/core/common_runtime/bfc_allocator.cc:654] 1 Chunks of size 865280000 totalling 825.20MiB 2018-10-19 17:53:13.158160: I tensorflow/core/common_runtime/bfc_allocator.cc:654] 1 Chunks of size 1474560000 totalling 1.37GiB 2018-10-19 17:53:13.158187: I tensorflow/core/common_runtime/bfc_allocator.cc:658] Sum Total of in-use chunks: 2.18GiB 2018-10-19 17:53:13.158233: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Stats: Limit: 3699351552 InUse: 2344715008 MaxInUse: 3203046400 NumAllocs: 95 MaxAllocSize: 1474560000

2018-10-19 17:53:13.158279: W tensorflow/core/common_runtime/bfc_allocator.cc:275] *___****____ 2018-10-19 17:53:13.158377: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at conv_ops.cc:693 : Resource exhausted: OOM when allocating tensor with shape[10000,64,24,24] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1292, in _do_call return fn(*args) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1277, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1367, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[10000,64,24,24] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node import/sequential_1/conv2d_2/convolution}} = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/sequential_1/conv2d_1/Relu, import/conv2d_2/kernel)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[{{node import/sequential_1/dense_2/Softmax/_7}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_54_import/sequential_1/dense_2/Softmax", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "tftrt_example.py", line 138, in main() File "tftrt_example.py", line 118, in main y_tf = tf_engine.infer(x_test) File "tftrt_example.py", line 50, in infer feed_dict={self.x_tensor: x}) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 887, in run run_metadata_ptr) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1110, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1286, in _do_run run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1308, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[10000,64,24,24] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node import/sequential_1/conv2d_2/convolution}} = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/sequential_1/conv2d_1/Relu, import/conv2d_2/kernel)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[{{node import/sequential_1/dense_2/Softmax/_7}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_54_import/sequential_1/dense_2/Softmax", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'import/sequential_1/conv2d_2/convolution', defined at: File "tftrt_example.py", line 138, in main() File "tftrt_example.py", line 116, in main tf_engine = TfEngine(frozen_graph) File "tftrt_example.py", line 38, in init graph_def=graph.frozen, return_elements=graph.x_name + graph.y_name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def _ProcessNewOps(graph) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3426, in _add_new_tf_operations for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3426, in for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3285, in _create_op_from_tf_operation ret = Operation(c_op, self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[10000,64,24,24] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node import/sequential_1/conv2d_2/convolution}} = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/sequential_1/conv2d_1/Relu, import/conv2d_2/kernel)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[{{node import/sequential_1/dense_2/Softmax/_7}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_54_import/sequential_1/dense_2/Softmax", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

jeng1220 commented 5 years ago

please file another issue, not this issue thread