search

tensorflow / tensorrt

TensorFlow/TensorRT integration
Apache License 2.0
736 stars 226 forks source link

DefaultLogger INVALID_ARGUMENT: Cannot set empty memory #273

Open ProtossDragoon opened 3 years ago

ProtossDragoon commented 3 years ago

TensorRT Version : 7.1.3 (nvidia container install) TensorFlow Version : 2.5 (nvidia container install) Platform : Jetson AGX Xavier Jetpack Version : 4.5.1

Hi, while using TF-TRT, always I faced on an error 2021-10-04 00:54:11.478005: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory. when I called converter.build(input_fn=my_input_fn) method. (either the error raised at runtime when not call build() method for pre-build engine.)

(image : trials, parameter tuning minimum_segment_size and so on.) image

But any of them couldn't remove that error message.

My entire source code wasn't different with any other example source codes.

#https://github.com/tensorflow/tensorrt/issues/195
#https://github.com/tensorflow/tensorrt/issues/200
env = os.environ.get('TF_DEBUG_TRT_ALLOW_INEFFICIENT_TRANSPOSE')
new_env = os.environ['TF_DEBUG_TRT_ALLOW_INEFFICIENT_TRANSPOSE'] = "1"
print(f'Setting environment variable... TF_DEBUG_TRT_ALLOW_INEFFICIENT_TRANSPOSE={new_env}... (before {env})')

# Conversion Parameters 
conversion_params = trt.TrtConversionParams(
    precision_mode=QUANTIZATION_MODE,
    # is_dynamic_op=False if IS_THERE_DYNAMIC_ELEMENT == 'No' else True,
    max_workspace_size_bytes=MAX_GPU_TRTENGINE_SIZE_MB<<20,
    minimum_segment_size=MINIMUM_SEGMENT_SIZE,
    maximum_cached_engines=MAXIMUM_CACHED_ENGINE,
    use_calibration=True # This argument is ignored if precision_mode is not INT8.
    )

converter = trt.TrtGraphConverterV2(
    input_saved_model_dir=SAVED_MODEL_DIR,
    input_saved_model_tags=SAVED_MODEL_TAG_SET,
    input_saved_model_signature_key=SAVED_MODEL_SIGNATURE_DEF,
    conversion_params=conversion_params
    )

if QUANTIZATION_MODE.lower() == 'int8':
    print('Convert with calibration')
    converter.convert(
        calibration_input_fn=lambda:input_fn(args) # args : my command line input
        ) # Maybe conberter.convert() works fine...?
else:
    print('Convert without calibration')
    converter.convert()

print('[Info] Convert complete.')
print('[Info] Now prebuild trt-engine.')
converter.build(input_fn=lambda:input_fn(args)) # here raises error message, but process doesn't shut down.

optimized_saved_model_fulldir = parse_tftrt_path(args)

# Save the model to the disk
converter.save(optimized_saved_model_fulldir)
print('[Info] writing savedmodel complete.')

Is that error message ignorable? How can I remove the error properly?

Thanks!



Entire program log description below

2021-10-07 12:58:32.487612: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-10-07 12:58:32.513042: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 12:58:32.513711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-10-07 12:58:32.514587: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
2021-10-07 12:58:32.515218: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.10
2021-10-07 12:58:32.515752: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.10
2021-10-07 12:58:32.516480: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-10-07 12:58:32.522466: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-10-07 12:58:32.535133: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.10
2021-10-07 12:58:32.543363: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.10
2021-10-07 12:58:32.546243: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-10-07 12:58:32.546781: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 12:58:32.547194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 12:58:32.547471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0

==============================
2021-10-07 12:58:34.474165: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
The given SavedModel contains the following tag-sets:
'serve'

2021-10-07 12:59:08.535907: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:
SignatureDef key: "__saved_model_init_op"
SignatureDef key: "serving_default"

2021-10-07 12:59:41.622920: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
The given SavedModel SignatureDef contains the following input(s):
  inputs['input_1'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, -1, -1, 3)
      name: serving_default_input_1:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['softmax'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, -1, -1, 9)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
==============================

Setting environment variable... TF_DEBUG_TRT_ALLOW_INEFFICIENT_TRANSPOSE=1... (before None)
2021-10-07 13:00:15.205272: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libnvinfer.so.7
Convert with calibration
2021-10-07 13:01:31.023349: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:31.023722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-10-07 13:01:31.024083: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:31.024338: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:31.024470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0
2021-10-07 13:01:31.025038: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
2021-10-07 13:01:36.768804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-07 13:01:36.768996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2021-10-07 13:01:36.769069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2021-10-07 13:01:36.769398: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:36.769747: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:36.769976: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:36.770180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18830 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2021-10-07 13:06:42.865817: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.866173: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-10-07 13:06:42.866754: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-07 13:06:42.870918: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.871261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-10-07 13:06:42.871593: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.871875: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.872000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0
2021-10-07 13:06:42.872134: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-07 13:06:42.872198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2021-10-07 13:06:42.872245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2021-10-07 13:06:42.872483: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.872836: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.873158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18830 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2021-10-07 13:06:42.874915: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 31250000 Hz
2021-10-07 13:06:47.236124: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1171] Optimization results for grappler item: graph_to_optimize
  function_optimizer: Graph size after: 1216 nodes (901), 1824 edges (1509), time = 176.228ms.
  function_optimizer: function_optimizer did nothing. time = 2.022ms.

2021-10-07 13:07:39.201871: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.202299: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-10-07 13:07:39.202699: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-07 13:07:39.210304: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.210650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-10-07 13:07:39.211093: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.211425: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.211551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0
2021-10-07 13:07:39.211696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-07 13:07:39.211766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2021-10-07 13:07:39.211813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2021-10-07 13:07:39.212127: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.212488: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.212722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18830 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2021-10-07 13:07:45.495807: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:790] There are 30 ops of 8 different types in the graph that are not converted to TensorRT: Identity, ResizeNearestNeighbor, Placeholder, NoOp, Mul, Shape, DataFormatVecPermute, StridedSlice, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2021-10-07 13:07:46.085000: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:759] Number of TensorRT candidate segments: 6
2021-10-07 13:07:46.227369: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 0 consisting of 424 nodes by TRTEngineOp_0_0.
2021-10-07 13:07:46.232840: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 1 consisting of 22 nodes by StatefulPartitionedCall/model/TRTEngineOp_0_1.
2021-10-07 13:07:46.234022: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 2 consisting of 22 nodes by StatefulPartitionedCall/model/TRTEngineOp_0_2.
2021-10-07 13:07:46.234935: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 3 consisting of 22 nodes by StatefulPartitionedCall/model/TRTEngineOp_0_3.
2021-10-07 13:07:46.236717: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 4 consisting of 22 nodes by StatefulPartitionedCall/model/TRTEngineOp_0_4.
2021-10-07 13:07:46.237427: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 5 consisting of 27 nodes by TRTEngineOp_0_5.
2021-10-07 13:07:47.073017: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1171] Optimization results for grappler item: tf_graph
  constant_folding: Graph size after: 558 nodes (-624), 1200 edges (-624), time = 554.18ms.
  layout: Graph size after: 595 nodes (37), 1237 edges (37), time = 328.333ms.
  constant_folding: Graph size after: 587 nodes (-8), 1229 edges (-8), time = 230.304ms.
  TensorRTOptimizer: Graph size after: 54 nodes (-533), 62 edges (-1167), time = 975.276ms.
  constant_folding: Graph size after: 54 nodes (0), 62 edges (0), time = 21.947ms.
Optimization results for grappler item: StatefulPartitionedCall/model/TRTEngineOp_0_1_native_segment
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 26.112ms.
  layout: Graph size after: 26 nodes (0), 25 edges (0), time = 42.376ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 31.746ms.
  TensorRTOptimizer: Graph size after: 26 nodes (0), 25 edges (0), time = 5.574ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 22.56ms.
Optimization results for grappler item: StatefulPartitionedCall/model/TRTEngineOp_0_2_native_segment
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 6.176ms.
  layout: Graph size after: 26 nodes (0), 25 edges (0), time = 9.286ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 8.636ms.
  TensorRTOptimizer: Graph size after: 26 nodes (0), 25 edges (0), time = 0.877ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 6.713ms.
Optimization results for grappler item: StatefulPartitionedCall/model/TRTEngineOp_0_4_native_segment
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.232ms.
  layout: Graph size after: 26 nodes (0), 25 edges (0), time = 4.541ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.265ms.
  TensorRTOptimizer: Graph size after: 26 nodes (0), 25 edges (0), time = 0.405ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.41ms.
Optimization results for grappler item: TRTEngineOp_0_5_native_segment
  constant_folding: Graph size after: 29 nodes (0), 40 edges (0), time = 4.414ms.
  layout: Graph size after: 29 nodes (0), 40 edges (0), time = 4.662ms.
  constant_folding: Graph size after: 29 nodes (0), 40 edges (0), time = 4.751ms.
  TensorRTOptimizer: Graph size after: 29 nodes (0), 40 edges (0), time = 0.367ms.
  constant_folding: Graph size after: 29 nodes (0), 40 edges (0), time = 4.544ms.
Optimization results for grappler item: StatefulPartitionedCall/model/TRTEngineOp_0_3_native_segment
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.687ms.
  layout: Graph size after: 26 nodes (0), 25 edges (0), time = 7.218ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.93ms.
  TensorRTOptimizer: Graph size after: 26 nodes (0), 25 edges (0), time = 0.535ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.577ms.
Optimization results for grappler item: TRTEngineOp_0_0_native_segment
  constant_folding: Graph size after: 431 nodes (0), 440 edges (0), time = 57.452ms.
  layout: Graph size after: 431 nodes (0), 440 edges (0), time = 88.399ms.
  constant_folding: Graph size after: 431 nodes (0), 440 edges (0), time = 70.943ms.
  TensorRTOptimizer: Graph size after: 431 nodes (0), 440 edges (0), time = 8.772ms.
  constant_folding: Graph size after: 431 nodes (0), 440 edges (0), time = 66.01ms.

Found 30 files belonging to 1 classes.
2021-10-07 13:40:34.221298: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
step 1/30, image shape : (1, 320, 320, 3)
2021-10-07 13:40:35.773429: I tensorflow/compiler/tf2tensorrt/common/utils.cc:58] Linked TensorRT version: 7.1.3
2021-10-07 13:40:35.773914: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libnvinfer.so.7
2021-10-07 13:40:35.773981: I tensorflow/compiler/tf2tensorrt/common/utils.cc:60] Loaded TensorRT version: 7.1.3
2021-10-07 13:40:35.785002: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libnvinfer_plugin.so.7
2021-10-07 13:41:56.956338: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-10-07 13:41:56.964140: I tensorflow/stream_executor/cuda/cuda_dnn.cc:380] Loaded cuDNN version 8000
2021-10-07 13:42:03.715476: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.10
step 2/30, image shape : (1, 320, 320, 3)
step 3/30, image shape : (1, 320, 320, 3)
step 4/30, image shape : (1, 320, 320, 3)
step 5/30, image shape : (1, 320, 320, 3)
step 6/30, image shape : (1, 320, 320, 3)
step 7/30, image shape : (1, 320, 320, 3)
step 8/30, image shape : (1, 320, 320, 3)
step 9/30, image shape : (1, 320, 320, 3)
step 10/30, image shape : (1, 320, 320, 3)
step 11/30, image shape : (1, 320, 320, 3)
step 12/30, image shape : (1, 320, 320, 3)
step 13/30, image shape : (1, 320, 320, 3)
step 14/30, image shape : (1, 320, 320, 3)
step 15/30, image shape : (1, 320, 320, 3)
step 16/30, image shape : (1, 320, 320, 3)
step 17/30, image shape : (1, 320, 320, 3)
step 18/30, image shape : (1, 320, 320, 3)
step 19/30, image shape : (1, 320, 320, 3)
step 20/30, image shape : (1, 320, 320, 3)
step 21/30, image shape : (1, 320, 320, 3)
step 22/30, image shape : (1, 320, 320, 3)
step 23/30, image shape : (1, 320, 320, 3)
step 24/30, image shape : (1, 320, 320, 3)
step 25/30, image shape : (1, 320, 320, 3)
step 26/30, image shape : (1, 320, 320, 3)
step 27/30, image shape : (1, 320, 320, 3)
step 28/30, image shape : (1, 320, 320, 3)
step 29/30, image shape : (1, 320, 320, 3)
step 30/30, image shape : (1, 320, 320, 3)
[Info] Convert complete.
[Info] Now prebuild trt-engine.
Found 30 files belonging to 1 classes.
step 1/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.501273: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.518895: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.520264: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.521836: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.523049: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.525338: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 2/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.553969: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.555540: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.556665: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.558038: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.559088: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.559983: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 3/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.594004: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.597923: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.599552: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.601033: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.602305: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.604312: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 4/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.634213: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.638674: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.640178: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.642291: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.643825: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.649495: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 5/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.668430: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.671339: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.672554: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.678498: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.679625: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.681399: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 6/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.699913: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.703150: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.705529: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.708076: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.711813: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.713090: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 7/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.748968: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.750285: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.751345: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.752453: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.753522: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.754501: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 8/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.773218: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.774877: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.776059: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.777162: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.778034: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.779149: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 9/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.794973: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.796168: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.797143: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.797970: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.798687: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.799433: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 10/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.848015: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.849429: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.850419: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.851397: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.852383: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.853300: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 11/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.890036: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.892058: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.905496: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.906838: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.907802: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.908979: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 12/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.931153: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.933730: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.935208: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.936988: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.938206: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.945270: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 13/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.996327: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.997990: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.999318: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.000395: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.001443: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.002548: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 14/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.026953: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.028467: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.029830: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.030897: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.033084: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.034140: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 15/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.049957: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.051390: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.052484: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.053757: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.054732: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.055674: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 16/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.072601: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.075016: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.077017: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.078947: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.080625: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.081977: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 17/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.122032: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.123231: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.124173: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.125291: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.126088: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.126915: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 18/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.142622: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.144997: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.146100: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.147124: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.147956: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.148818: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 19/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.163239: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.165280: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.167056: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.170588: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.173496: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.174742: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 20/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.191675: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.193147: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.195249: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.196687: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.197986: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.198907: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 21/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.223086: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.224427: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.225536: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.226596: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.227501: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.228680: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 22/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.243688: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.245667: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.247469: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.253290: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.255746: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.258155: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 23/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.272156: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.273568: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.274988: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.276136: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.277656: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.278708: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 24/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.299802: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.301459: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.303199: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.304482: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.305795: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.306835: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 25/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.322027: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.324808: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.326183: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.327298: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.328234: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.329430: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 26/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.347076: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.348356: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.351511: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.354776: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.357402: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.358476: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 27/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.378031: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.383370: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.386487: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.387798: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.393018: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.395249: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 28/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.426818: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.431675: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.434095: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.436594: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.437868: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.438853: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 29/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.454118: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.455327: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.456274: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.460399: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.461846: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.462862: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 30/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.475968: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.477193: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.478811: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.479840: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.480687: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.481721: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
[Info] writing savedmodel complete.