Failed to 'convert conformer encoder with layernorm plugin'

Describe the bug

The script run_streaming_small_model.sh failed at stage 2

To Reproduce

Follow https://github.com/wenet-e2e/wenet/tree/main/runtime/gpu/tensorrt

docker pull soar97/triton-wenet:22.12
docker run -it --rm --name "wenet_trt_test" --gpus all --shm-size 1g --net host soar97/triton-wenet:22.12
git clone https://github.com/wenet-e2e/wenet.git
pip3 install -r requirements.txt

cd wenet/runtime/gpu/tensorrt
pip3 install -r requirements.txt

pip3 install nvidia-pyindex

bash run_streaming_small_model.sh

Expected behavior

Succeed to run the script: run_streaming_small_model.sh

Screenshots

Desktop (please complete the following information):

OS: ubuntu 20.04
Version 20.04

Additional context

logs:

repalce onnx ops with layernorm plugin 
[I] RUNNING | Command: /usr/local/bin/polygraphy surgeon sanitize /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/u2pp_aishell2_onnx/encoder_fp16.onnx --fold-constant -o /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV2.onnx
[I] Loading model: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/u2pp_aishell2_onnx/encoder_fp16.onnx
[I] Original Model:
    Name: torch_jit | ONNX Opset: 14

    ---- 6 Graph Input(s) ----
    {chunk_xs [dtype=float16, shape=('B', 67, 80)],
     chunk_lens [dtype=int32, shape=('B',)],
     offset [dtype=int64, shape=('B', 1)],
     att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 8 Graph Output(s) ----
    {log_probs [dtype=float16, shape=('B', 16, 10)],
     log_probs_idx [dtype=int64, shape=('B', 16, 10)],
     chunk_out [dtype=float16, shape=('B', 16, 256)],
     chunk_out_lens [dtype=int32, shape=('B',)],
     r_offset [dtype=int64, shape=('B', 1)],
     r_att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     r_cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     r_cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 456 Initializer(s) ----

    ---- 2653 Node(s) ----

[I] Folding Constants | Pass 1
2023-12-08 09:03:20.556508006 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /embed/Unsqueeze_2
2023-12-08 09:03:20.556545117 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /Unsqueeze_2
[I]     Total Nodes | Original:  2653, After Folding:  1810 |   843 Nodes Folded
[I] Folding Constants | Pass 2
[I]     Total Nodes | Original:  1810, After Folding:  1810 |     0 Nodes Folded
[I] Saving ONNX model to: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV2.onnx
[I] New Model:
    Name: torch_jit | ONNX Opset: 14

    ---- 6 Graph Input(s) ----
    {chunk_xs [dtype=float16, shape=('B', 67, 80)],
     chunk_lens [dtype=int32, shape=('B',)],
     offset [dtype=int64, shape=('B', 1)],
     att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 8 Graph Output(s) ----
    {log_probs [dtype=float16, shape=('B', 16, 10)],
     log_probs_idx [dtype=int64, shape=('B', 16, 10)],
     chunk_out [dtype=float16, shape=('B', 16, 256)],
     chunk_out_lens [dtype=int32, shape=('B',)],
     r_offset [dtype=int64, shape=('B', 1)],
     r_att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     r_cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     r_cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 1168 Initializer(s) ----

    ---- 1810 Node(s) ----

[I] PASSED | Runtime: 4.426s | Command: /usr/local/bin/polygraphy surgeon sanitize /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/u2pp_aishell2_onnx/encoder_fp16.onnx --fold-constant -o /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV2.onnx
[W] Found distinct tensors that share the same name:
[id: 140059744119440] Variable (chunk_out): (shape=['B', 16, 256], dtype=float16)
[id: 140059741412032] Variable (chunk_out): (shape=[], dtype=<class 'numpy.float16'>)
Note: Producer node(s) of first tensor:
[/after_norm/Add_1 (Add)
    Inputs: [
        Variable (/after_norm/Mul_output_0): (shape=['B', 16, 256], dtype=float16)
        Constant (encoder.after_norm.bias): (shape=(256,), dtype=float16)
    ]
    Outputs: [
        Variable (chunk_out): (shape=['B', 16, 256], dtype=float16)
    ]]
Producer node(s) of second tensor:
[LayerNormN-72 (LayerNorm)
    Inputs: [
        Variable (LayerNormV-71): (shape=None, dtype=float32)
        Constant (LayerNormB-72): (shape=(256,), dtype=float32)
        Constant (LayerNormA-72): (shape=(256,), dtype=float32)
    ]
    Outputs: [
        Variable (chunk_out): (shape=[], dtype=<class 'numpy.float16'>)
    ]]
[W] Found distinct tensors that share the same name:
[id: 140059744119440] Variable (chunk_out): (shape=['B', 16, 256], dtype=float16)
[id: 140059741412032] Variable (chunk_out): (shape=[], dtype=<class 'numpy.float16'>)
Note: Producer node(s) of first tensor:
[]
Producer node(s) of second tensor:
[LayerNormN-72 (LayerNorm)
    Inputs: [
        Variable (LayerNormV-71): (shape=None, dtype=float32)
        Constant (LayerNormB-72): (shape=(256,), dtype=float32)
        Constant (LayerNormA-72): (shape=(256,), dtype=float32)
    ]
    Outputs: [
        Variable (chunk_out): (shape=[], dtype=<class 'numpy.float16'>)
    ]]
[I] RUNNING | Command: /usr/local/bin/polygraphy surgeon sanitize /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV3.onnx --fold-constant -o /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
[I] Loading model: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV3.onnx
[W] Falling back to `onnx.shape_inference` because `onnxruntime.tools.symbolic_shape_infer` either could not be loaded or did not run successfully.
    Note that using ONNX-Runtime for shape inference may be faster and require less memory.
    Consider installing ONNX-Runtime or setting POLYGRAPHY_AUTOINSTALL_DEPS=1 in your environment variables to allow Polygraphy to do so automatically.
[W] ONNX shape inference exited with an error:
    [ShapeInferenceError] (op_type:MatMul, node name: /encoders.0/feed_forward_macaron/w_1/MatMul): [TypeInferenceError] Inferred elem type differs from existing elem type: (1) vs (10)
[I] Loading model: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV3.onnx
[I] Original Model:
    Name: torch_jit | ONNX Opset: 14

    ---- 6 Graph Input(s) ----
    {chunk_xs [dtype=float16, shape=('B', 67, 80)],
     chunk_lens [dtype=int32, shape=('B',)],
     offset [dtype=int64, shape=('B', 1)],
     att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 8 Graph Output(s) ----
    {log_probs [dtype=float16, shape=('B', 16, 10)],
     log_probs_idx [dtype=int64, shape=('B', 16, 10)],
     chunk_out [dtype=float16, shape=()],
     chunk_out_lens [dtype=int32, shape=('B',)],
     r_offset [dtype=int64, shape=('B', 1)],
     r_att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     r_cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     r_cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 1022 Initializer(s) ----

    ---- 1226 Node(s) ----

[I] Folding Constants | Pass 1
[W] ONNX shape inference exited with an error:
    [ShapeInferenceError] (op_type:MatMul, node name: /encoders.0/feed_forward_macaron/w_1/MatMul): [TypeInferenceError] Inferred elem type differs from existing elem type: (1) vs (10)
[I]     Total Nodes | Original:  1226, After Folding:  1226 |     0 Nodes Folded
[W] ONNX shape inference exited with an error:
    [ShapeInferenceError] (op_type:MatMul, node name: /encoders.0/feed_forward_macaron/w_1/MatMul): [TypeInferenceError] Inferred elem type differs from existing elem type: (1) vs (10)
[I] Saving ONNX model to: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
[I] New Model:
    Name: torch_jit | ONNX Opset: 14

    ---- 6 Graph Input(s) ----
    {chunk_xs [dtype=float16, shape=('B', 67, 80)],
     chunk_lens [dtype=int32, shape=('B',)],
     offset [dtype=int64, shape=('B', 1)],
     att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 8 Graph Output(s) ----
    {log_probs [dtype=float16, shape=('B', 16, 10)],
     log_probs_idx [dtype=int64, shape=('B', 16, 10)],
     chunk_out [dtype=float16, shape=()],
     chunk_out_lens [dtype=int32, shape=('B',)],
     r_offset [dtype=int64, shape=('B', 1)],
     r_att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     r_cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     r_cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 1022 Initializer(s) ----

    ---- 1226 Node(s) ----

[I] PASSED | Runtime: 3.356s | Command: /usr/local/bin/polygraphy surgeon sanitize /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV3.onnx --fold-constant -o /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
convert conformer encoder with layernorm plugin
&&&& RUNNING TensorRT.trtexec [TensorRT v8501] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=/mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx --minShapes=chunk_xs:1x67x80,chunk_lens:1,offset:1x1,att_cache:1x12x4x80x128,cnn_cache:1x12x256x7,cache_mask:1x1x80 --optShapes=chunk_xs:16x67x80,chunk_lens:16,offset:16x1,att_cache:16x12x4x80x128,cnn_cache:16x12x256x7,cache_mask:16x1x80 --maxShapes=chunk_xs:32x67x80,chunk_lens:32,offset:32x1,att_cache:32x12x4x80x128,cnn_cache:32x12x256x7,cache_mask:32x1x80 --plugins=/mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so --saveEngine=/mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoder_fp16.plan
[12/08/2023-09:03:27] [I] === Model Options ===
[12/08/2023-09:03:27] [I] Format: ONNX
[12/08/2023-09:03:27] [I] Model: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
[12/08/2023-09:03:27] [I] Output:
[12/08/2023-09:03:27] [I] === Build Options ===
[12/08/2023-09:03:27] [I] Max batch: explicit batch
[12/08/2023-09:03:27] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[12/08/2023-09:03:27] [I] minTiming: 1
[12/08/2023-09:03:27] [I] avgTiming: 8
[12/08/2023-09:03:27] [I] Precision: FP32+FP16
[12/08/2023-09:03:27] [I] LayerPrecisions: 
[12/08/2023-09:03:27] [I] Calibration: 
[12/08/2023-09:03:27] [I] Refit: Disabled
[12/08/2023-09:03:27] [I] Sparsity: Disabled
[12/08/2023-09:03:27] [I] Safe mode: Disabled
[12/08/2023-09:03:27] [I] DirectIO mode: Disabled
[12/08/2023-09:03:27] [I] Restricted mode: Disabled
[12/08/2023-09:03:27] [I] Build only: Disabled
[12/08/2023-09:03:27] [I] Save engine: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoder_fp16.plan
[12/08/2023-09:03:27] [I] Load engine: 
[12/08/2023-09:03:27] [I] Profiling verbosity: 0
[12/08/2023-09:03:27] [I] Tactic sources: Using default tactic sources
[12/08/2023-09:03:27] [I] timingCacheMode: local
[12/08/2023-09:03:27] [I] timingCacheFile: 
[12/08/2023-09:03:27] [I] Heuristic: Disabled
[12/08/2023-09:03:27] [I] Preview Features: Use default preview flags.
[12/08/2023-09:03:27] [I] Input(s)s format: fp32:CHW
[12/08/2023-09:03:27] [I] Output(s)s format: fp32:CHW
[12/08/2023-09:03:27] [I] Input build shape: chunk_xs=1x67x80+16x67x80+32x67x80
[12/08/2023-09:03:27] [I] Input build shape: chunk_lens=1+16+32
[12/08/2023-09:03:27] [I] Input build shape: cache_mask=1x1x80+16x1x80+32x1x80
[12/08/2023-09:03:27] [I] Input build shape: offset=1x1+16x1+32x1
[12/08/2023-09:03:27] [I] Input build shape: cnn_cache=1x12x256x7+16x12x256x7+32x12x256x7
[12/08/2023-09:03:27] [I] Input build shape: att_cache=1x12x4x80x128+16x12x4x80x128+32x12x4x80x128
[12/08/2023-09:03:27] [I] Input calibration shapes: model
[12/08/2023-09:03:27] [I] === System Options ===
[12/08/2023-09:03:27] [I] Device: 0
[12/08/2023-09:03:27] [I] DLACore: 
[12/08/2023-09:03:27] [I] Plugins: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so
[12/08/2023-09:03:27] [I] === Inference Options ===
[12/08/2023-09:03:27] [I] Batch: Explicit
[12/08/2023-09:03:27] [I] Input inference shape: att_cache=16x12x4x80x128
[12/08/2023-09:03:27] [I] Input inference shape: cnn_cache=16x12x256x7
[12/08/2023-09:03:27] [I] Input inference shape: offset=16x1
[12/08/2023-09:03:27] [I] Input inference shape: cache_mask=16x1x80
[12/08/2023-09:03:27] [I] Input inference shape: chunk_lens=16
[12/08/2023-09:03:27] [I] Input inference shape: chunk_xs=16x67x80
[12/08/2023-09:03:27] [I] Iterations: 10
[12/08/2023-09:03:27] [I] Duration: 3s (+ 200ms warm up)
[12/08/2023-09:03:27] [I] Sleep time: 0ms
[12/08/2023-09:03:27] [I] Idle time: 0ms
[12/08/2023-09:03:27] [I] Streams: 1
[12/08/2023-09:03:27] [I] ExposeDMA: Disabled
[12/08/2023-09:03:27] [I] Data transfers: Enabled
[12/08/2023-09:03:27] [I] Spin-wait: Disabled
[12/08/2023-09:03:27] [I] Multithreading: Disabled
[12/08/2023-09:03:27] [I] CUDA Graph: Disabled
[12/08/2023-09:03:27] [I] Separate profiling: Disabled
[12/08/2023-09:03:27] [I] Time Deserialize: Disabled
[12/08/2023-09:03:27] [I] Time Refit: Disabled
[12/08/2023-09:03:27] [I] NVTX verbosity: 0
[12/08/2023-09:03:27] [I] Persistent Cache Ratio: 0
[12/08/2023-09:03:27] [I] Inputs:
[12/08/2023-09:03:27] [I] === Reporting Options ===
[12/08/2023-09:03:27] [I] Verbose: Disabled
[12/08/2023-09:03:27] [I] Averages: 10 inferences
[12/08/2023-09:03:27] [I] Percentiles: 90,95,99
[12/08/2023-09:03:27] [I] Dump refittable layers:Disabled
[12/08/2023-09:03:27] [I] Dump output: Disabled
[12/08/2023-09:03:27] [I] Profile: Disabled
[12/08/2023-09:03:27] [I] Export timing to JSON file: 
[12/08/2023-09:03:27] [I] Export output to JSON file: 
[12/08/2023-09:03:27] [I] Export profile to JSON file: 
[12/08/2023-09:03:27] [I] 
[12/08/2023-09:03:27] [I] === Device Information ===
[12/08/2023-09:03:27] [I] Selected Device: NVIDIA GeForce GTX 1080 Ti
[12/08/2023-09:03:27] [I] Compute Capability: 6.1
[12/08/2023-09:03:27] [I] SMs: 28
[12/08/2023-09:03:27] [I] Compute Clock Rate: 1.645 GHz
[12/08/2023-09:03:27] [I] Device Global Memory: 11170 MiB
[12/08/2023-09:03:27] [I] Shared Memory per SM: 96 KiB
[12/08/2023-09:03:27] [I] Memory Bus Width: 352 bits (ECC disabled)
[12/08/2023-09:03:27] [I] Memory Clock Rate: 5.505 GHz
[12/08/2023-09:03:27] [I] 
[12/08/2023-09:03:27] [I] TensorRT version: 8.5.1
[12/08/2023-09:03:27] [I] Loading supplied plugin library: /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so
[12/08/2023-09:03:27] [I] [TRT] [MemUsageChange] Init CUDA: CPU +9, GPU +0, now: CPU 22, GPU 144 (MiB)
[12/08/2023-09:03:29] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +122, GPU +22, now: CPU 199, GPU 166 (MiB)
[12/08/2023-09:03:29] [I] Start parsing network model
[12/08/2023-09:03:29] [I] [TRT] ----------------------------------------------------------------
[12/08/2023-09:03:29] [I] [TRT] Input filename:   /mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
[12/08/2023-09:03:29] [I] [TRT] ONNX IR version:  0.0.8
[12/08/2023-09:03:29] [I] [TRT] Opset version:    14
[12/08/2023-09:03:29] [I] [TRT] Producer name:    pytorch
[12/08/2023-09:03:29] [I] [TRT] Producer version: 1.13.0
[12/08/2023-09:03:29] [I] [TRT] Domain:           
[12/08/2023-09:03:29] [I] [TRT] Model version:    0
[12/08/2023-09:03:29] [I] [TRT] Doc string:       
[12/08/2023-09:03:29] [I] [TRT] ----------------------------------------------------------------
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/08/2023-09:03:29] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:29] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:29] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:29] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:29] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:29] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:29] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:29] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:29] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:29] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:30] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:30] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:30] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:30] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:31] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:31] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:31] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:31] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:32] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:32] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:32] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:32] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:32] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:32] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:32] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:32] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:32] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:32] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:32] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:32] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:32] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:32] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:32] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:32] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:32] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:32] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:32] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:32] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:32] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:32] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:32] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:32] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:33] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:33] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:33] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:33] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:33] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:33] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:33] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:33] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:33] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:33] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:33] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:33] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:33] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:33] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:33] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:33] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:33] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:33] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:33] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:33] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:33] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:33] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:33] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:33] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:34] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:34] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:34] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:34] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:34] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:34] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:34] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:34] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:34] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:34] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:34] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:34] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:34] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:34] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:34] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:34] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:34] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:34] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:34] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:34] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:34] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:34] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:34] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:34] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:35] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:35] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:35] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:35] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:35] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:35] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:35] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:35] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:35] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:35] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:35] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:35] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:35] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:35] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:35] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:35] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:35] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:35] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:35] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:35] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:36] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:36] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:36] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:36] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:36] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:36] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:36] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:36] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:37] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:37] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:37] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:37] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:37] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:37] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:37] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:37] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:37] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:37] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:37] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:37] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:37] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:37] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:37] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:37] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:37] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:37] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:37] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:37] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:38] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:38] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:38] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:38] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:38] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:38] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:38] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:38] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:39] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:39] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:39] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:39] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:39] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:39] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:39] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:39] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:39] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:39] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:39] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:39] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:39] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:39] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:39] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:39] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:40] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:40] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:40] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:40] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:40] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:40] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:40] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:40] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:40] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:40] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:40] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:40] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:41] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:41] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:41] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:41] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:41] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/08/2023-09:03:41] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/08/2023-09:03:41] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/08/2023-09:03:41] [I] [TRT] Successfully created plugin: LayerNorm
[12/08/2023-09:03:41] [I] Finish parsing network model
[12/08/2023-09:03:45] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +12, now: CPU 340, GPU 178 (MiB)
[12/08/2023-09:03:45] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 341, GPU 188 (MiB)
[12/08/2023-09:03:45] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[12/08/2023-09:04:42] [E] Error[2]: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
[12/08/2023-09:04:42] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[12/08/2023-09:04:42] [E] Engine could not be created from network
[12/08/2023-09:04:42] [E] Building engine failed
[12/08/2023-09:04:42] [E] Failed to create engine from model or file.
[12/08/2023-09:04:42] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8501] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=/mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx --minShapes=chunk_xs:1x67x80,chunk_lens:1,offset:1x1,att_cache:1x12x4x80x128,cnn_cache:1x12x256x7,cache_mask:1x1x80 --optShapes=chunk_xs:16x67x80,chunk_lens:16,offset:16x1,att_cache:16x12x4x80x128,cnn_cache:16x12x256x7,cache_mask:16x1x80 --maxShapes=chunk_xs:32x67x80,chunk_lens:32,offset:32x1,att_cache:32x12x4x80x128,cnn_cache:32x12x256x7,cache_mask:32x1x80 --plugins=/mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so --saveEngine=/mnt/xt/8T/CODES/Audio/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoder_fp16.plan

Could you try export with '--verbose' and attached detailed log?
Or I suggest to use latest tensorrt ( >= 8.6), and onnx opset e.g. 17, since it would support layernorm plugin internally. So, we could skip the stage https://github.com/wenet-e2e/wenet/blob/main/runtime/gpu/tensorrt/run_streaming_small_model.sh#L56-L64 and remove '--plugin' option.

@yuekaizhang Thank you for your replay. I achieved my aim by following https://github.com/wenet-e2e/wenet/blob/main/runtime/gpu/README.md. If any update on docs about using tensorrt as backend, I will try again.

@yuekaizhang could you plz take a look at this log? I tried trtexec with --verbose

[12/19/2023-12:45:27] [V] [TRT] Casting to type: float16
[12/19/2023-12:45:27] [V] [TRT] Registering layer: /Cast_7 for ONNX node: /Cast_7
[12/19/2023-12:45:27] [V] [TRT] Registering tensor: log_probs_1131 for ONNX tensor: log_probs
[12/19/2023-12:45:27] [V] [TRT] /Cast_7 [Cast] outputs: [log_probs -> (-1, 16, 10)[HALF]], 
[12/19/2023-12:45:27] [V] [TRT] Marking log_probs_1131 as output: log_probs
[12/19/2023-12:45:27] [V] [TRT] Marking log_probs_idx_1130 as output: log_probs_idx
[12/19/2023-12:45:27] [V] [TRT] Marking chunk_out_1125 as output: chunk_out
[12/19/2023-12:45:27] [V] [TRT] Marking chunk_out_lens_1126 as output: chunk_out_lens
[12/19/2023-12:45:27] [V] [TRT] Marking r_offset_1129 as output: r_offset
[12/19/2023-12:45:27] [V] [TRT] Marking r_att_cache_1123 as output: r_att_cache
[12/19/2023-12:45:27] [V] [TRT] Marking r_cnn_cache_1124 as output: r_cnn_cache
[12/19/2023-12:45:27] [V] [TRT] Marking r_cache_mask_534 as output: r_cache_mask
[12/19/2023-12:45:27] [I] Finish parsing network model
[12/19/2023-12:45:27] [E] Error[4]: [network.cpp::validate::3076] Error Code 4: Internal Error (cnn_cache: for dimension number 3 in profile 0 does not match network definition (got min=7, opt=7, max=7), expected min=opt=max=14).)
[12/19/2023-12:45:27] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[12/19/2023-12:45:27] [E] Engine could not be created from network
[12/19/2023-12:45:27] [E] Building engine failed
[12/19/2023-12:45:27] [E] Failed to create engine from model or file.
[12/19/2023-12:45:27] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8501] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/encoderV4.onnx --minShapes=chunk_xs:1x67x80,chunk_lens:1,offset:1x1,att_cache:1x12x4x80x128,cnn_cache:1x12x256x7,cache_mask:1x1x80 --optShapes=chunk_xs:16x67x80,chunk_lens:16,offset:16x1,att_cache:16x12x4x80x128,cnn_cache:16x12x256x7,cache_mask:16x1x80 --maxShapes=chunk_xs:32x67x80,chunk_lens:32,offset:32x1,att_cache:32x12x4x80x128,cnn_cache:32x12x256x7,cache_mask:32x1x80 --plugins=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/LayerNorm.so --saveEngine=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/encoder_fp16.plan --verbose

"Error Code 4: Internal Error (cnn_cache: for dimension number 3 in profile 0 does not match network definition (got min=7, opt=7, max=7), expected min=opt=max=14).)"

Looks like you are using conv filter with 31, rather than 15, please change 7 with 14 for cnn_cache for below codes,

/usr/src/tensorrt/bin/trtexec --fp16 --onnx=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/encoderV4.onnx --minShapes=chunk_xs:1x67x80,chunk_lens:1,offset:1x1,att_cache:1x12x4x80x128,cnn_cache:1x12x256x7,cache_mask:1x1x80 --optShapes=chunk_xs:16x67x80,chunk_lens:16,offset:16x1,att_cache:16x12x4x80x128,cnn_cache:16x12x256x7,cache_mask:16x1x80 --maxShapes=chunk_xs:32x67x80,chunk_lens:32,offset:32x1,att_cache:32x12x4x80x128,cnn_cache:32x12x256x7,cache_mask:32x1x80 --plugins=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/LayerNorm.so --saveEngine=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/encoder_fp16.plan --verbose

Thank you for your reply.

I tried again with 14 instead of 7. It made more progress, but still failed.

[12/20/2023-12:40:54] [V] [TRT] Tactic: 0x000000000000001c Time: 0.00356148
[12/20/2023-12:40:54] [V] [TRT] Tactic: 0x000000000000001d Time: 0.00377095
[12/20/2023-12:40:54] [V] [TRT] Fastest Tactic: 0x000000000000000f Time: 0.00324706
[12/20/2023-12:40:54] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWise)
[12/20/2023-12:40:54] [V] [TRT] PointWise has no valid tactics for this config, skipping
[12/20/2023-12:40:54] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PointWiseV2 Tactic: 0x000000000000000f
[12/20/2023-12:40:54] [V] [TRT] =============== Computing costs for 
[12/20/2023-12:40:54] [V] [TRT] *************** Autotuning format combination: Float(4096,256,1), Float(1), Float(1) -> Float(4096,256,1) ***************
[12/20/2023-12:40:54] [V] [TRT] --------------- Timing Runner: LayerNormN-0 (PluginV2)
[12/20/2023-12:40:54] [V] [TRT] Deleting timing cache: 845 entries, served 18643 hits since creation.
[12/20/2023-12:40:54] [E] Error[2]: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
[12/20/2023-12:40:54] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[12/20/2023-12:40:54] [E] Engine could not be created from network
[12/20/2023-12:40:54] [E] Building engine failed
[12/20/2023-12:40:54] [E] Failed to create engine from model or file.
[12/20/2023-12:40:54] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8501] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/encoderV4.onnx --minShapes=chunk_xs:1x67x80,chunk_lens:1,offset:1x1,att_cache:1x12x4x80x128,cnn_cache:1x12x256x14,cache_mask:1x1x80 --optShapes=chunk_xs:16x67x80,chunk_lens:16,offset:16x1,att_cache:16x12x4x80x128,cnn_cache:16x12x256x14,cache_mask:16x1x80 --maxShapes=chunk_xs:32x67x80,chunk_lens:32,offset:32x1,att_cache:32x12x4x80x128,cnn_cache:32x12x256x14,cache_mask:32x1x80 --plugins=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/LayerNorm.so --saveEngine=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet//models/multi_cn_unified_conformer_exp/exp_streaming_trt/encoder_fp16.plan --verbose

The steps to reproduce

clone the repo, and the commit I used is 1f8e1c51f4f685282b30077642c701837265925f
download the pre-trained model: multi_cn_unified_conformer_exp.tar.gz

convert to onnx

model_dir=${PWD}/models/multi_cn_unified_conformer_exp
onnx_model_dir=${model_dir}/onnx/
mkdir $onnx_model_dir

python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn --ctc_weight=0.5 --output_onnx_dir=$onnx_model_dir --fp16 --streaming
cp $model_dir/train.yaml $onnx_model_dir/
#cp $model_dir/words.txt $onnx_model_dir/
cp $model_dir/units.txt $onnx_model_dir/words.txt

tensorrt test

docker pull soar97/triton-wenet:22.12
docker run -it --rm --name "wenet_trt_test" --gpus all -v /mnt/xt/8T:/mnt/xt/8T/ --shm-size 1g --net host soar97/triton-wenet:22.12

# inside the docker container
WENET_DIR=/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/
onnx_model_dir=${WENET_DIR}/models/multi_cn_unified_conformer_exp/onnx
outputs_dir=${WENET_DIR}/models/multi_cn_unified_conformer_exp/exp_streaming_trt
mkdir $outputs_dir
model_repo_path=${WENET_DIR}/runtime/gpu/tensorrt/model_repo_stateful_trt

# compile LayerNormPlugin
cd ${WENET_DIR}/runtime/gpu/tensorrt/LayerNormPlugin
make clean
make
cp LayerNorm.so $outputs_dir
cd ..

# repalce onnx ops with layernorm plugin 
polygraphy surgeon sanitize $onnx_model_dir/encoder_fp16.onnx --fold-constant -o $outputs_dir/encoderV2.onnx
python3 replace_layernorm.py --input_onnx $outputs_dir/encoderV2.onnx \
                           --output_onnx $outputs_dir/encoderV3.onnx
polygraphy surgeon sanitize $outputs_dir/encoderV3.onnx --fold-constant -o $outputs_dir/encoderV4.onnx

# convert conformer encoder with layernorm plugin
MIN_BATCH=1
OPT_BATCH=16
MAX_BATCH=32
/usr/src/tensorrt/bin/trtexec \
  --fp16 \
  --onnx=$outputs_dir/encoderV4.onnx \
  --minShapes=chunk_xs:${MIN_BATCH}x67x80,chunk_lens:${MIN_BATCH},offset:${MIN_BATCH}x1,att_cache:${MIN_BATCH}x12x4x80x128,cnn_cache:${MIN_BATCH}x12x256x14,cache_mask:${MIN_BATCH}x1x80 \
  --optShapes=chunk_xs:${OPT_BATCH}x67x80,chunk_lens:${OPT_BATCH},offset:${OPT_BATCH}x1,att_cache:${OPT_BATCH}x12x4x80x128,cnn_cache:${OPT_BATCH}x12x256x14,cache_mask:${OPT_BATCH}x1x80 \
  --maxShapes=chunk_xs:${MAX_BATCH}x67x80,chunk_lens:${MAX_BATCH},offset:${MAX_BATCH}x1,att_cache:${MAX_BATCH}x12x4x80x128,cnn_cache:${MAX_BATCH}x12x256x14,cache_mask:${MAX_BATCH}x1x80 \
  --plugins=$outputs_dir/LayerNorm.so \
  --saveEngine=$outputs_dir/encoder_fp16.plan \
  --verbose

Would you mind trying https://github.com/wenet-e2e/wenet/blob/3eb9a8579d65a32b606aa04c89bdfcaca10d220b/runtime/gpu/tensorrt/run_streaming_small_model.sh first? I am not sure if multi_cn_unified_conformer_exp.tar.gz works.

Also, I suggest to use latest tensorrt without layernorm plugin.

Or I suggest to use latest tensorrt ( >= 8.6), and onnx opset e.g. 17, since it would support layernorm plugin internally. So, we could skip the stage https://github.com/wenet-e2e/wenet/blob/main/runtime/gpu/tensorrt/run_streaming_small_model.sh#L56-L64 and remove '--plugin' option.

What's your gpu's name, btw?

Thank you. I'll try run_streaming_small_model.sh in few days.

I cannot use tensorrt>= 8.6, because my target device is a jetson nx board, with JetPack 5.1.2 only includes TensorRT 8.5.2.

I run the scripts in x86 PC with a 1080Ti gpu.

onnx

using http://mobvoi-speech-public.ufile.ucloud.cn/public/wenet/aishell2/20210618_u2pp_conformer_exp.tar.gz

repalce onnx ops with layernorm plugin

root@ubt:/mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/LayerNormPlugin# polygraphy surgeon sanitize $onnx_model_dir/encoder_fp16.onnx --fold-constant -o $outputs_dir/encoderV2.onnx
[W] 'colored' module is not installed, will not use colors when logging. To enable colors, please install the 'colored' module: python3 -m pip install colored
[I] RUNNING | Command: /usr/local/bin/polygraphy surgeon sanitize /mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/models/20210618_u2pp_conformer_exp/onnx//encoder_fp16.onnx --fold-constant -o /mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV2.onnx
[I] Loading model: /mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/models/20210618_u2pp_conformer_exp/onnx/encoder_fp16.onnx
[I] Original Model:
    Name: torch_jit | ONNX Opset: 14

    ---- 6 Graph Input(s) ----
    {chunk_xs [dtype=float16, shape=('B', 67, 80)],
     chunk_lens [dtype=int32, shape=('B',)],
     offset [dtype=int64, shape=('B', 1)],
     att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 8 Graph Output(s) ----
    {log_probs [dtype=float16, shape=('B', 16, 10)],
     log_probs_idx [dtype=int64, shape=('B', 16, 10)],
     chunk_out [dtype=float16, shape=('B', 16, 256)],
     chunk_out_lens [dtype=int32, shape=('B',)],
     r_offset [dtype=int64, shape=('B', 1)],
     r_att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     r_cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     r_cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 456 Initializer(s) ----

    ---- 2653 Node(s) ----

[I] Folding Constants | Pass 1
2023-12-21 12:44:50.040462676 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /embed/Unsqueeze_2
2023-12-21 12:44:50.040493041 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /Unsqueeze_2
[I]     Total Nodes | Original:  2653, After Folding:  1810 |   843 Nodes Folded
[I] Folding Constants | Pass 2
[I]     Total Nodes | Original:  1810, After Folding:  1810 |     0 Nodes Folded
[I] Saving ONNX model to: /mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV2.onnx
[I] New Model:
    Name: torch_jit | ONNX Opset: 14

    ---- 6 Graph Input(s) ----
    {chunk_xs [dtype=float16, shape=('B', 67, 80)],
     chunk_lens [dtype=int32, shape=('B',)],
     offset [dtype=int64, shape=('B', 1)],
     att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 8 Graph Output(s) ----
    {log_probs [dtype=float16, shape=('B', 16, 10)],
     log_probs_idx [dtype=int64, shape=('B', 16, 10)],
     chunk_out [dtype=float16, shape=('B', 16, 256)],
     chunk_out_lens [dtype=int32, shape=('B',)],
     r_offset [dtype=int64, shape=('B', 1)],
     r_att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     r_cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     r_cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 1168 Initializer(s) ----

    ---- 1810 Node(s) ----

[I] PASSED | Runtime: 4.063s | Command: /usr/local/bin/polygraphy surgeon sanitize /mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/models/20210618_u2pp_conformer_exp/onnx//encoder_fp16.onnx --fold-constant -o /mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV2.onnx

root@ubt://mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt# python3 replace_layernorm.py --input_onnx $outputs_dir/encoderV2.onnx \
>                                --output_onnx $outputs_dir/encoderV3.onnx \
>                                || exit 1
[W] colored module is not installed, will not use colors when logging. To enable colors, please install the colored module: python3 -m pip install colored
[W] Found distinct tensors that share the same name:
[id: 140050566716096] Variable (chunk_out): (shape=['B', 16, 256], dtype=float16)
[id: 140050564008688] Variable (chunk_out): (shape=[], dtype=<class 'numpy.float16'>)
Note: Producer node(s) of first tensor:
[/after_norm/Add_1 (Add)
    Inputs: [
        Variable (/after_norm/Mul_output_0): (shape=['B', 16, 256], dtype=float16)
        Constant (encoder.after_norm.bias): (shape=(256,), dtype=float16)
    ]
    Outputs: [
        Variable (chunk_out): (shape=['B', 16, 256], dtype=float16)
    ]]
Producer node(s) of second tensor:
[LayerNormN-72 (LayerNorm)
    Inputs: [
        Variable (LayerNormV-71): (shape=None, dtype=float32)
        Constant (LayerNormB-72): (shape=(256,), dtype=float32)
        Constant (LayerNormA-72): (shape=(256,), dtype=float32)
    ]
    Outputs: [
        Variable (chunk_out): (shape=[], dtype=<class 'numpy.float16'>)
    ]]
[W] colored module is not installed, will not use colors when logging. To enable colors, please install the colored module: python3 -m pip install colored
[W] Found distinct tensors that share the same name:
[id: 140050566716096] Variable (chunk_out): (shape=['B', 16, 256], dtype=float16)
[id: 140050564008688] Variable (chunk_out): (shape=[], dtype=<class 'numpy.float16'>)
Note: Producer node(s) of first tensor:
[]
Producer node(s) of second tensor:
[LayerNormN-72 (LayerNorm)
    Inputs: [
        Variable (LayerNormV-71): (shape=None, dtype=float32)
        Constant (LayerNormB-72): (shape=(256,), dtype=float32)
        Constant (LayerNormA-72): (shape=(256,), dtype=float32)
    ]
    Outputs: [
        Variable (chunk_out): (shape=[], dtype=<class 'numpy.float16'>)
    ]]

root@ubt://mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt# polygraphy surgeon sanitize $outputs_dir/encoderV3.onnx --fold-constant -o $outputs_dir/encoderV4.onnx
[W] 'colored' module is not installed, will not use colors when logging. To enable colors, please install the 'colored' module: python3 -m pip install colored
[I] RUNNING | Command: /usr/local/bin/polygraphy surgeon sanitize //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV3.onnx --fold-constant -o //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
[I] Loading model: //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV3.onnx
[W] Falling back to `onnx.shape_inference` because `onnxruntime.tools.symbolic_shape_infer` either could not be loaded or did not run successfully.
    Note that using ONNX-Runtime for shape inference may be faster and require less memory.
    Consider installing ONNX-Runtime or setting POLYGRAPHY_AUTOINSTALL_DEPS=1 in your environment variables to allow Polygraphy to do so automatically.
[W] ONNX shape inference exited with an error:
    [ShapeInferenceError] (op_type:MatMul, node name: /encoders.0/feed_forward_macaron/w_1/MatMul): [TypeInferenceError] Inferred elem type differs from existing elem type: (1) vs (10)
[I] Loading model: //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV3.onnx
[I] Original Model:
    Name: torch_jit | ONNX Opset: 14

    ---- 6 Graph Input(s) ----
    {chunk_xs [dtype=float16, shape=('B', 67, 80)],
     chunk_lens [dtype=int32, shape=('B',)],
     offset [dtype=int64, shape=('B', 1)],
     att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 8 Graph Output(s) ----
    {log_probs [dtype=float16, shape=('B', 16, 10)],
     log_probs_idx [dtype=int64, shape=('B', 16, 10)],
     chunk_out [dtype=float16, shape=()],
     chunk_out_lens [dtype=int32, shape=('B',)],
     r_offset [dtype=int64, shape=('B', 1)],
     r_att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     r_cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     r_cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 1022 Initializer(s) ----

    ---- 1226 Node(s) ----

[I] Folding Constants | Pass 1
[W] ONNX shape inference exited with an error:
    [ShapeInferenceError] (op_type:MatMul, node name: /encoders.0/feed_forward_macaron/w_1/MatMul): [TypeInferenceError] Inferred elem type differs from existing elem type: (1) vs (10)
[I]     Total Nodes | Original:  1226, After Folding:  1226 |     0 Nodes Folded
[W] ONNX shape inference exited with an error:
    [ShapeInferenceError] (op_type:MatMul, node name: /encoders.0/feed_forward_macaron/w_1/MatMul): [TypeInferenceError] Inferred elem type differs from existing elem type: (1) vs (10)
[I] Saving ONNX model to: //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
[I] New Model:
    Name: torch_jit | ONNX Opset: 14

    ---- 6 Graph Input(s) ----
    {chunk_xs [dtype=float16, shape=('B', 67, 80)],
     chunk_lens [dtype=int32, shape=('B',)],
     offset [dtype=int64, shape=('B', 1)],
     att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 8 Graph Output(s) ----
    {log_probs [dtype=float16, shape=('B', 16, 10)],
     log_probs_idx [dtype=int64, shape=('B', 16, 10)],
     chunk_out [dtype=float16, shape=()],
     chunk_out_lens [dtype=int32, shape=('B',)],
     r_offset [dtype=int64, shape=('B', 1)],
     r_att_cache [dtype=float16, shape=('B', 12, 4, 80, 128)],
     r_cnn_cache [dtype=float16, shape=('B', 12, 256, 7)],
     r_cache_mask [dtype=float16, shape=('B', 1, 80)]}

    ---- 1022 Initializer(s) ----

    ---- 1226 Node(s) ----

[I] PASSED | Runtime: 3.078s | Command: /usr/local/bin/polygraphy surgeon sanitize //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV3.onnx --fold-constant -o //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx

convert conformer encoder with layernorm plugin

      $trtexec \
>           --fp16 \
>           --onnx=$outputs_dir/encoderV4.onnx \
>           --minShapes=chunk_xs:${MIN_BATCH}x67x80,chunk_lens:${MIN_BATCH},offset:${MIN_BATCH}x1,att_cache:${MIN_BATCH}x12x4x80x128,cnn_cache:${MIN_BATCH}x12x256x7,cache_mask:${MIN_BATCH}x1x80 \
>           --optShapes=chunk_xs:${OPT_BATCH}x67x80,chunk_lens:${OPT_BATCH},offset:${OPT_BATCH}x1,att_cache:${OPT_BATCH}x12x4x80x128,cnn_cache:${OPT_BATCH}x12x256x7,cache_mask:${OPT_BATCH}x1x80 \
>           --maxShapes=chunk_xs:${MAX_BATCH}x67x80,chunk_lens:${MAX_BATCH},offset:${MAX_BATCH}x1,att_cache:${MAX_BATCH}x12x4x80x128,cnn_cache:${MAX_BATCH}x12x256x7,cache_mask:${MAX_BATCH}x1x80 \
>           --plugins=$outputs_dir/LayerNorm.so \
>           --saveEngine=$outputs_dir/encoder_fp16.plan
&&&& RUNNING TensorRT.trtexec [TensorRT v8501] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx --minShapes=chunk_xs:1x67x80,chunk_lens:1,offset:1x1,att_cache:1x12x4x80x128,cnn_cache:1x12x256x7,cache_mask:1x1x80 --optShapes=chunk_xs:16x67x80,chunk_lens:16,offset:16x1,att_cache:16x12x4x80x128,cnn_cache:16x12x256x7,cache_mask:16x1x80 --maxShapes=chunk_xs:32x67x80,chunk_lens:32,offset:32x1,att_cache:32x12x4x80x128,cnn_cache:32x12x256x7,cache_mask:32x1x80 --plugins=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so --saveEngine=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoder_fp16.plan
[12/21/2023-12:57:49] [I] === Model Options ===
[12/21/2023-12:57:49] [I] Format: ONNX
[12/21/2023-12:57:49] [I] Model: //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
[12/21/2023-12:57:49] [I] Output:
[12/21/2023-12:57:49] [I] === Build Options ===
[12/21/2023-12:57:49] [I] Max batch: explicit batch
[12/21/2023-12:57:49] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[12/21/2023-12:57:49] [I] minTiming: 1
[12/21/2023-12:57:49] [I] avgTiming: 8
[12/21/2023-12:57:49] [I] Precision: FP32+FP16
[12/21/2023-12:57:49] [I] LayerPrecisions: 
[12/21/2023-12:57:49] [I] Calibration: 
[12/21/2023-12:57:49] [I] Refit: Disabled
[12/21/2023-12:57:49] [I] Sparsity: Disabled
[12/21/2023-12:57:49] [I] Safe mode: Disabled
[12/21/2023-12:57:49] [I] DirectIO mode: Disabled
[12/21/2023-12:57:49] [I] Restricted mode: Disabled
[12/21/2023-12:57:49] [I] Build only: Disabled
[12/21/2023-12:57:49] [I] Save engine: //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoder_fp16.plan
[12/21/2023-12:57:49] [I] Load engine: 
[12/21/2023-12:57:49] [I] Profiling verbosity: 0
[12/21/2023-12:57:49] [I] Tactic sources: Using default tactic sources
[12/21/2023-12:57:49] [I] timingCacheMode: local
[12/21/2023-12:57:49] [I] timingCacheFile: 
[12/21/2023-12:57:49] [I] Heuristic: Disabled
[12/21/2023-12:57:49] [I] Preview Features: Use default preview flags.
[12/21/2023-12:57:49] [I] Input(s)s format: fp32:CHW
[12/21/2023-12:57:49] [I] Output(s)s format: fp32:CHW
[12/21/2023-12:57:49] [I] Input build shape: chunk_xs=1x67x80+16x67x80+32x67x80
[12/21/2023-12:57:49] [I] Input build shape: chunk_lens=1+16+32
[12/21/2023-12:57:49] [I] Input build shape: cache_mask=1x1x80+16x1x80+32x1x80
[12/21/2023-12:57:49] [I] Input build shape: offset=1x1+16x1+32x1
[12/21/2023-12:57:49] [I] Input build shape: cnn_cache=1x12x256x7+16x12x256x7+32x12x256x7
[12/21/2023-12:57:49] [I] Input build shape: att_cache=1x12x4x80x128+16x12x4x80x128+32x12x4x80x128
[12/21/2023-12:57:49] [I] Input calibration shapes: model
[12/21/2023-12:57:49] [I] === System Options ===
[12/21/2023-12:57:49] [I] Device: 0
[12/21/2023-12:57:49] [I] DLACore: 
[12/21/2023-12:57:49] [I] Plugins: //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so
[12/21/2023-12:57:49] [I] === Inference Options ===
[12/21/2023-12:57:49] [I] Batch: Explicit
[12/21/2023-12:57:49] [I] Input inference shape: att_cache=16x12x4x80x128
[12/21/2023-12:57:49] [I] Input inference shape: cnn_cache=16x12x256x7
[12/21/2023-12:57:49] [I] Input inference shape: offset=16x1
[12/21/2023-12:57:49] [I] Input inference shape: cache_mask=16x1x80
[12/21/2023-12:57:49] [I] Input inference shape: chunk_lens=16
[12/21/2023-12:57:49] [I] Input inference shape: chunk_xs=16x67x80
[12/21/2023-12:57:49] [I] Iterations: 10
[12/21/2023-12:57:49] [I] Duration: 3s (+ 200ms warm up)
[12/21/2023-12:57:49] [I] Sleep time: 0ms
[12/21/2023-12:57:49] [I] Idle time: 0ms
[12/21/2023-12:57:49] [I] Streams: 1
[12/21/2023-12:57:49] [I] ExposeDMA: Disabled
[12/21/2023-12:57:49] [I] Data transfers: Enabled
[12/21/2023-12:57:49] [I] Spin-wait: Disabled
[12/21/2023-12:57:49] [I] Multithreading: Disabled
[12/21/2023-12:57:49] [I] CUDA Graph: Disabled
[12/21/2023-12:57:49] [I] Separate profiling: Disabled
[12/21/2023-12:57:49] [I] Time Deserialize: Disabled
[12/21/2023-12:57:49] [I] Time Refit: Disabled
[12/21/2023-12:57:49] [I] NVTX verbosity: 0
[12/21/2023-12:57:49] [I] Persistent Cache Ratio: 0
[12/21/2023-12:57:49] [I] Inputs:
[12/21/2023-12:57:49] [I] === Reporting Options ===
[12/21/2023-12:57:49] [I] Verbose: Disabled
[12/21/2023-12:57:49] [I] Averages: 10 inferences
[12/21/2023-12:57:49] [I] Percentiles: 90,95,99
[12/21/2023-12:57:49] [I] Dump refittable layers:Disabled
[12/21/2023-12:57:49] [I] Dump output: Disabled
[12/21/2023-12:57:49] [I] Profile: Disabled
[12/21/2023-12:57:49] [I] Export timing to JSON file: 
[12/21/2023-12:57:49] [I] Export output to JSON file: 
[12/21/2023-12:57:49] [I] Export profile to JSON file: 
[12/21/2023-12:57:49] [I] 
[12/21/2023-12:57:49] [I] === Device Information ===
[12/21/2023-12:57:49] [I] Selected Device: NVIDIA GeForce GTX 1080 Ti
[12/21/2023-12:57:49] [I] Compute Capability: 6.1
[12/21/2023-12:57:49] [I] SMs: 28
[12/21/2023-12:57:49] [I] Compute Clock Rate: 1.645 GHz
[12/21/2023-12:57:49] [I] Device Global Memory: 11170 MiB
[12/21/2023-12:57:49] [I] Shared Memory per SM: 96 KiB
[12/21/2023-12:57:49] [I] Memory Bus Width: 352 bits (ECC disabled)
[12/21/2023-12:57:49] [I] Memory Clock Rate: 5.505 GHz
[12/21/2023-12:57:49] [I] 
[12/21/2023-12:57:49] [I] TensorRT version: 8.5.1
[12/21/2023-12:57:49] [I] Loading supplied plugin library: //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so
[12/21/2023-12:57:49] [I] [TRT] [MemUsageChange] Init CUDA: CPU +9, GPU +0, now: CPU 22, GPU 149 (MiB)
[12/21/2023-12:57:51] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +122, GPU +22, now: CPU 199, GPU 171 (MiB)
[12/21/2023-12:57:51] [I] Start parsing network model
[12/21/2023-12:57:51] [I] [TRT] ----------------------------------------------------------------
[12/21/2023-12:57:51] [I] [TRT] Input filename:   //mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx
[12/21/2023-12:57:51] [I] [TRT] ONNX IR version:  0.0.8
[12/21/2023-12:57:51] [I] [TRT] Opset version:    14
[12/21/2023-12:57:51] [I] [TRT] Producer name:    pytorch
[12/21/2023-12:57:51] [I] [TRT] Producer version: 1.13.0
[12/21/2023-12:57:51] [I] [TRT] Domain:           
[12/21/2023-12:57:51] [I] [TRT] Model version:    0
[12/21/2023-12:57:51] [I] [TRT] Doc string:       
[12/21/2023-12:57:51] [I] [TRT] ----------------------------------------------------------------
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:51] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:51] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:51] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:52] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:52] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:52] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:52] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:53] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:53] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:53] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:53] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:54] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:54] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:54] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:54] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:54] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:54] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:54] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:54] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:54] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:54] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:54] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:54] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:54] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:54] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:54] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:54] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:54] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:54] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:54] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:54] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:54] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:54] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:54] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:54] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:54] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:54] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:54] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:54] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:54] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:54] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:54] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:54] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:55] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:55] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:55] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:55] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:55] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:55] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:55] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:55] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:55] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:55] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:55] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:55] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:55] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:55] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:55] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:55] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:55] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:55] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:55] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:55] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:55] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:55] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:55] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:55] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:55] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:55] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:55] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:55] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:56] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:56] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:56] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:56] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:56] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:56] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:56] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:56] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:56] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:56] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:56] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:56] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:56] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:56] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:56] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:56] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:56] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:56] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:56] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:56] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:57] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:57] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:57] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:57] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:57] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:57] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:57] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:57] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:57] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:57] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:57] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:57] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:57] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:57] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:57] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:57] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:58] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:58] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:58] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:58] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:58] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:58] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:58] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:58] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:58] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:58] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:58] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:58] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:58] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:58] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:58] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:58] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:59] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:59] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:59] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:59] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:59] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:59] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:59] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:59] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:59] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:59] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:59] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:59] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:59] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:59] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:59] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:59] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:57:59] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:57:59] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:57:59] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:57:59] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:58:00] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:58:00] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:58:00] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:58:00] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:58:00] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:58:00] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:58:00] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:58:00] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:58:00] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:58:00] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:58:00] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:58:00] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:58:00] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:58:00] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:58:00] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:58:00] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:58:00] [I] [TRT] No importer registered for op: LayerNorm. Attempting to import as plugin.
[12/21/2023-12:58:00] [I] [TRT] Searching for plugin: LayerNorm, plugin_version: 1, plugin_namespace: 
[12/21/2023-12:58:00] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[12/21/2023-12:58:00] [I] [TRT] Successfully created plugin: LayerNorm
[12/21/2023-12:58:01] [I] Finish parsing network model
[12/21/2023-12:58:04] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +12, now: CPU 340, GPU 183 (MiB)
[12/21/2023-12:58:04] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 341, GPU 193 (MiB)
[12/21/2023-12:58:04] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[12/21/2023-12:58:57] [E] Error[2]: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
[12/21/2023-12:58:57] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[12/21/2023-12:58:57] [E] Engine could not be created from network
[12/21/2023-12:58:57] [E] Building engine failed
[12/21/2023-12:58:57] [E] Failed to create engine from model or file.
[12/21/2023-12:58:57] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8501] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx --minShapes=chunk_xs:1x67x80,chunk_lens:1,offset:1x1,att_cache:1x12x4x80x128,cnn_cache:1x12x256x7,cache_mask:1x1x80 --optShapes=chunk_xs:16x67x80,chunk_lens:16,offset:16x1,att_cache:16x12x4x80x128,cnn_cache:16x12x256x7,cache_mask:16x1x80 --maxShapes=chunk_xs:32x67x80,chunk_lens:32,offset:32x1,att_cache:32x12x4x80x128,cnn_cache:32x12x256x7,cache_mask:32x1x80 --plugins=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so --saveEngine=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoder_fp16.plan

run `trtexec` with `--verbose` (only show the last the part of the log)

[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x00000000020a0543 numSplitK: 5 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x00000000020a0543 Time: 0.194414
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x00000000020c0543 numSplitK: 6 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x00000000020c0543 Time: 0.176713
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x0000000002100543 numSplitK: 8 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002100543 Time: 0.195291
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x0000000002180543 numSplitK: 12 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002180543 Time: 0.189001
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x0000000002200543 numSplitK: 16 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002200543 Time: 0.19968
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x0000000002300543 numSplitK: 24 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002300543 Time: 0.188709
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x0000000002400543 numSplitK: 32 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002400543 Time: 0.189294
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x0000000002600543 numSplitK: 48 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002600543 Time: 0.253659
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_128x128_relu_tn_v1 Tactic: 0x0000000002800543 numSplitK: 64 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002800543 Time: 0.312027
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000000020555 numSplitK: 1 numBuffers: 0 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000000020555 Time: 0.397312
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002040555 numSplitK: 2 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002040555 Time: 0.245614
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002060555 numSplitK: 3 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002060555 Time: 0.205678
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002080555 numSplitK: 4 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002080555 Time: 0.197362
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x00000000020a0555 numSplitK: 5 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x00000000020a0555 Time: 0.188123
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x00000000020c0555 numSplitK: 6 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x00000000020c0555 Time: 0.172178
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002100555 numSplitK: 8 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002100555 Time: 0.189147
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002180555 numSplitK: 12 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002180555 Time: 0.196946
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002200555 numSplitK: 16 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002200555 Time: 0.197193
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002300555 numSplitK: 24 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002300555 Time: 0.201024
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002400555 numSplitK: 32 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002400555 Time: 0.205358
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002600555 numSplitK: 48 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002600555 Time: 0.246053
[12/21/2023-13:05:20] [V] [TRT] Set Tactic Name: maxwell_sgemm_64x64_relu_tn_v1 Tactic: 0x0000000002800555 numSplitK: 64 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0000000002800555 Time: 0.29301
[12/21/2023-13:05:20] [V] [TRT] Fastest Tactic: 0x00000000020c02fb Time: 0.107593
[12/21/2023-13:05:20] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskFlattenConvolution)
[12/21/2023-13:05:20] [V] [TRT] CaskFlattenConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:20] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskConvolution)
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x32_relu_small_nn_v0 Tactic: 0x632674f65e3422ae
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x632674f65e3422ae Time: 0.582071
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v0 Tactic: 0xfac2e123a5eb1714
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xfac2e123a5eb1714 Time: 0.595529
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x128_relu_medium_nn_v1 Tactic: 0x3e787008e11a6129
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x3e787008e11a6129 Time: 0.578853
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x32_relu_interior_nn_v0 Tactic: 0x1792ed6b0f1ea883
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x1792ed6b0f1ea883 Time: 0.553623
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x64_relu_medium_nn_v0 Tactic: 0xef1674e9526bef07
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xef1674e9526bef07 Time: 0.651995
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x32_relu_medium_nn_v1 Tactic: 0x0ebe499388e08286
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x0ebe499388e08286 Time: 0.530432
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x64_relu_small_nn_v0 Tactic: 0xf462d2631d68e4d5
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xf462d2631d68e4d5 Time: 0.631077
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x128_relu_small_nn_v0 Tactic: 0x4963fb96b4067e81
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x4963fb96b4067e81 Time: 0.595383
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x128_relu_small_nn_v1 Tactic: 0xfa4db728b7a121ee
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xfa4db728b7a121ee Time: 0.568759
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v0 Tactic: 0x4c5584586319b832
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x4c5584586319b832 Time: 0.611182
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x64_relu_medium_nn_v1 Tactic: 0x5c38385751ccb068
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x5c38385751ccb068 Time: 0.498834
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x64_relu_small_nn_v1 Tactic: 0x474c9edd1ecfbbba
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x474c9edd1ecfbbba Time: 0.486107
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x32_relu_interior_nn_v1 Tactic: 0xa4bca1d50cb9f7ec
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xa4bca1d50cb9f7ec Time: 0.508343
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x32_relu_small_nn_v1 Tactic: 0xd00838485d937dc1
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xd00838485d937dc1 Time: 0.512731
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x32_relu_medium_nn_v0 Tactic: 0xbd90052d8b47dde9
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xbd90052d8b47dde9 Time: 0.582217
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v1 Tactic: 0x49ecad9da64c487b
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x49ecad9da64c487b Time: 0.562322
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v1 Tactic: 0xff7bc8e660bee75d
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xff7bc8e660bee75d Time: 0.485376
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x128_relu_medium_nn_v0 Tactic: 0x8d563cb6e2bd3e46
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x8d563cb6e2bd3e46 Time: 0.595675
[12/21/2023-13:05:20] [V] [TRT] Fastest Tactic: 0xff7bc8e660bee75d Time: 0.485376
[12/21/2023-13:05:20] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CublasConvolution Tactic: 0x0000000000000001
[12/21/2023-13:05:20] [V] [TRT] *************** Autotuning format combination: Float(4864,1,4864,4864) -> Float(256,1,256,256) ***************
[12/21/2023-13:05:20] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CublasConvolution)
[12/21/2023-13:05:20] [V] [TRT] CublasConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:20] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskFlattenConvolution)
[12/21/2023-13:05:20] [V] [TRT] CaskFlattenConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:20] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskConvolution)
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x64_sliced1x2_ldg4_relu_exp_small_nhwc_tn_v1 Tactic: 0x9961ac24fc07a1df
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x9961ac24fc07a1df Time: 0.231278
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x32_sliced1x4_ldg4_relu_exp_small_nhwc_tn_v1 Tactic: 0x80f932c0b8ce5940
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x80f932c0b8ce5940 Time: 0.116517
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x32x8_stage1_warpsize2x2x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x3b4f6ab5a921947c
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x3b4f6ab5a921947c Time: 0.429495
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x32x8_stage1_warpsize1x2x1_g1_ffma_aligna4_alignc4 Tactic: 0xc317ae77d87da5d4
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xc317ae77d87da5d4 Time: 0.397312
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x64x8_stage1_warpsize1x4x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x139f635dfeb9045f
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x139f635dfeb9045f Time: 0.412818
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x32x8_stage1_warpsize1x2x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x9cafecb2124e5252
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x9cafecb2124e5252 Time: 0.307931
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x64x8_stage1_warpsize1x2x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0xe93f2e28f98fc61d
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0xe93f2e28f98fc61d Time: 0.415013
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x128x8_stage1_warpsize1x4x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0x3bebf50cd5b7caf3
[12/21/2023-13:05:20] [V] [TRT] Tactic: 0x3bebf50cd5b7caf3 Time: 0.624187
[12/21/2023-13:05:20] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x256x8_stage1_warpsize1x4x1_g1_ffma_aligna4_alignc4 Tactic: 0x401a7ef58abeade7
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x401a7ef58abeade7 Time: 2.22574
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize256x64x8_stage1_warpsize2x2x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x5eafbc1f35548767
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x5eafbc1f35548767 Time: 1.62962
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x64_sliced1x2_ldg4_relu_exp_interior_nhwc_tn_v1 Tactic: 0x35f071de80e3b3c4
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x35f071de80e3b3c4 Time: 0.236544
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x64x8_stage1_warpsize2x2x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0x1e3b9c0eb36f2d9e
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x1e3b9c0eb36f2d9e Time: 0.557349
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x64x8_stage1_warpsize1x2x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x901a66ea74c49078
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x901a66ea74c49078 Time: 0.406875
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x128x8_stage1_warpsize2x2x1_g1_ffma_aligna4_alignc4 Tactic: 0xf39be0e2a13bbf04
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xf39be0e2a13bbf04 Time: 2.38651
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x128x8_stage1_warpsize1x4x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x2517a9de7106ad9e
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x2517a9de7106ad9e Time: 0.576366
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x64x8_stage1_warpsize2x2x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x37fae0edcfab5656
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x37fae0edcfab5656 Time: 0.529147
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x32x8_stage1_warpsize1x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x0b5f8a8e0c0f34e3
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x0b5f8a8e0c0f34e3 Time: 0.361618
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x128x8_stage1_warpsize1x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x5684c2d732a13190
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x5684c2d732a13190 Time: 0.632247
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x128x8_stage1_warpsize2x2x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0x5954f9135918ec0e
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x5954f9135918ec0e Time: 2.38182
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x128x8_stage1_warpsize2x2x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x2c23155378d4d1c0
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x2c23155378d4d1c0 Time: 2.38124
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize256x64x8_stage1_warpsize2x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x795d7b676cc5a93f
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x795d7b676cc5a93f Time: 1.92293
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize256x64x8_stage1_warpsize2x2x1_g1_ffma_aligna4_alignc4 Tactic: 0xc3436966322cd072
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xc3436966322cd072 Time: 1.71432
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize256x64x8_stage1_warpsize2x2x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0x63b292bd77b3a5bd
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x63b292bd77b3a5bd Time: 1.65625
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_scudnn_128x32_sliced1x4_ldg4_relu_exp_interior_nhwc_tn_v1 Tactic: 0x5c024b37b78e77c0
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x5c024b37b78e77c0 Time: 0.123099
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x64x8_stage1_warpsize1x4x1_g1_ffma_aligna4_alignc4 Tactic: 0x18fd395d26b83e59
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x18fd395d26b83e59 Time: 0.453339
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x64x8_stage1_warpsize2x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x5efe566ea68f6f0c
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x5efe566ea68f6f0c Time: 0.574025
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x256x8_stage1_warpsize1x4x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0xd5baa07cd2fa38e8
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xd5baa07cd2fa38e8 Time: 2.23086
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x128x8_stage1_warpsize1x4x1_g1_ffma_aligna4_alignc4 Tactic: 0x16828257a071ae34
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x16828257a071ae34 Time: 0.630199
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x32x8_stage1_warpsize1x2x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0xc2883ad6ccb77226
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xc2883ad6ccb77226 Time: 0.406235
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x32x8_stage1_warpsize2x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x382d34906ae01028
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x382d34906ae01028 Time: 0.473088
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x64x8_stage1_warpsize1x4x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0xa39445a71180bf67
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xa39445a71180bf67 Time: 0.43403
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x128x8_stage1_warpsize1x4x1_g1_ffma_aligna4_alignc4 Tactic: 0xac9c9056fe98d779
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xac9c9056fe98d779 Time: 0.651086
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x16x8_stage1_warpsize4x1x1_g1_ffma_aligna4_alignc4 Tactic: 0xd1ba7ae831f85905
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xd1ba7ae831f85905 Time: 0.368494
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x32x8_stage1_warpsize1x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x9e1ddb1c7167a2bc
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x9e1ddb1c7167a2bc Time: 0.466505
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x32x8_stage1_warpsize1x2x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0xc69395bd46c55685
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xc69395bd46c55685 Time: 0.413403
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x256x8_stage1_warpsize1x4x1_g1_ffma_aligna4_alignc4 Tactic: 0xfa046cf4d457d4aa
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xfa046cf4d457d4aa Time: 2.24005
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x128x8_stage1_warpsize2x2x1_g1_ffma_aligna4_alignc4 Tactic: 0xa8c8dd0948906ce6
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xa8c8dd0948906ce6 Time: 2.18744
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x64x8_stage1_warpsize1x2x1_g1_ffma_aligna4_alignc4 Tactic: 0xa5c4cc891412daf0
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xa5c4cc891412daf0 Time: 0.465189
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x64x8_stage1_warpsize2x2x1_g1_ffma_aligna4_alignc4 Tactic: 0xe4e0446ff8661641
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xe4e0446ff8661641 Time: 0.657408
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x32x8_stage1_warpsize2x2x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0x996492d082c67c95
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x996492d082c67c95 Time: 0.432567
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x32x8_stage1_warpsize2x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x8233269134096965
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x8233269134096965 Time: 0.508489
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x32x8_stage1_warpsize1x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x5655ffe5a515338b
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x5655ffe5a515338b Time: 0.425426
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x128x8_stage1_warpsize1x2x1_g1_ffma_aligna4_alignc4 Tactic: 0xec9ad0d66c4848dd
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xec9ad0d66c4848dd Time: 0.841289
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize128x16x8_stage1_warpsize4x1x1_g1_ffma_aligna4_alignc4 Tactic: 0x6ba468e96f112048
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x6ba468e96f112048 Time: 0.42101
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x32x8_stage1_warpsize1x2x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0x6e6020f6c8269716
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x6e6020f6c8269716 Time: 0.341778
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x128x8_stage1_warpsize1x2x1_g1_ffma_t1r1s1_aligna4_alignc4 Tactic: 0x2a465439d58607e4
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x2a465439d58607e4 Time: 0.621714
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x64x8_stage1_warpsize1x2x1_g1_ffma_aligna4_alignc4 Tactic: 0x6d8ce870c0604bc7
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x6d8ce870c0604bc7 Time: 0.45173
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x256x8_stage1_warpsize1x4x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x242042bde1014f5f
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x242042bde1014f5f Time: 2.19911
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_indexed_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x64x8_stage1_warpsize1x4x1_g1_ffma_aligna4_alignc4 Tactic: 0xd0b51da4f2caaf6e
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0xd0b51da4f2caaf6e Time: 0.461531
[12/21/2023-13:05:21] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: sm50_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize32x128x8_stage1_warpsize1x2x1_g1_ffma_simple_t1r1s1_aligna4_alignc4 Tactic: 0x612156dda64593a4
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x612156dda64593a4 Time: 0.623031
[12/21/2023-13:05:21] [V] [TRT] Fastest Tactic: 0x80f932c0b8ce5940 Time: 0.116517
[12/21/2023-13:05:21] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CaskConvolution Tactic: 0x80f932c0b8ce5940
[12/21/2023-13:05:21] [V] [TRT] *************** Autotuning format combination: Float(1216,1:4,1216,1216) -> Float(64,1:4,64,64) ***************
[12/21/2023-13:05:21] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CudaDepthwiseConvolution)
[12/21/2023-13:05:21] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:21] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CublasConvolution)
[12/21/2023-13:05:21] [V] [TRT] CublasConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:21] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskGemmConvolution)
[12/21/2023-13:05:21] [V] [TRT] CaskGemmConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:21] [V] [TRT] *************** Autotuning format combination: Half(4864,1,1,1) -> Half(256,1,1,1) ***************
[12/21/2023-13:05:21] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CudnnConvolution)
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.392338
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.354889
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x0000000000000002 Time: 0.328704
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x0000000000000004 Time: 39.4132
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x0000000000000005 Time: 5.10874
[12/21/2023-13:05:21] [V] [TRT] Fastest Tactic: 0x0000000000000002 Time: 0.328704
[12/21/2023-13:05:21] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CublasConvolution)
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.0964754
[12/21/2023-13:05:21] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.0907703
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x0000000000000004 Time: 4.53925
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x0000000000000005 Time: 4.53032
[12/21/2023-13:05:22] [V] [TRT] Fastest Tactic: 0x0000000000000001 Time: 0.0907703
[12/21/2023-13:05:22] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskGemmConvolution)
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000002000d numSplitK: 1 numBuffers: 0 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000002000d Time: 20.8253
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000204000d numSplitK: 2 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000204000d Time: 10.4458
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000206000d numSplitK: 3 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000206000d Time: 7.00489
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000208000d numSplitK: 4 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000208000d Time: 5.2676
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x00000000020a000d numSplitK: 5 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x00000000020a000d Time: 4.26145
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x00000000020c000d numSplitK: 6 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x00000000020c000d Time: 3.58429
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000210000d numSplitK: 8 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000210000d Time: 5.32334
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000218000d numSplitK: 12 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000218000d Time: 3.66096
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000220000d numSplitK: 16 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000220000d Time: 4.1355
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000230000d numSplitK: 24 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000230000d Time: 3.90671
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000240000d numSplitK: 32 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000240000d Time: 3.65875
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000260000d numSplitK: 48 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000260000d Time: 3.58546
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_tn_v0 Tactic: 0x000000000280000d numSplitK: 64 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000280000d Time: 3.8046
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000002017d numSplitK: 1 numBuffers: 0 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000002017d Time: 20.8154
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000204017d numSplitK: 2 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:22] [V] [TRT] Tactic: 0x000000000204017d Time: 10.4435
[12/21/2023-13:05:22] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000206017d numSplitK: 3 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000206017d Time: 7.00372
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000208017d numSplitK: 4 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000208017d Time: 5.27389
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x00000000020a017d numSplitK: 5 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x00000000020a017d Time: 4.26057
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x00000000020c017d numSplitK: 6 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x00000000020c017d Time: 3.59658
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000210017d numSplitK: 8 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000210017d Time: 5.33299
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000218017d numSplitK: 12 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000218017d Time: 3.68464
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000220017d numSplitK: 16 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000220017d Time: 4.16095
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000230017d numSplitK: 24 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000230017d Time: 3.94757
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000240017d numSplitK: 32 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000240017d Time: 3.67104
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000260017d numSplitK: 48 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000260017d Time: 3.60229
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_nn_v0 Tactic: 0x000000000280017d numSplitK: 64 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000280017d Time: 3.81831
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000002025a numSplitK: 1 numBuffers: 0 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000002025a Time: 20.8156
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000204025a numSplitK: 2 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000204025a Time: 10.4428
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000206025a numSplitK: 3 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000206025a Time: 7.00401
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000208025a numSplitK: 4 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000208025a Time: 5.27419
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x00000000020a025a numSplitK: 5 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x00000000020a025a Time: 4.26043
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x00000000020c025a numSplitK: 6 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x00000000020c025a Time: 3.59585
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000210025a numSplitK: 8 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000210025a Time: 5.33314
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000218025a numSplitK: 12 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000218025a Time: 3.68655
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000220025a numSplitK: 16 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000220025a Time: 4.16154
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000230025a numSplitK: 24 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000230025a Time: 3.94752
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000240025a numSplitK: 32 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000240025a Time: 3.67163
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000260025a numSplitK: 48 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:23] [V] [TRT] Tactic: 0x000000000260025a Time: 3.60492
[12/21/2023-13:05:23] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x128_tn_v0 Tactic: 0x000000000280025a numSplitK: 64 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000280025a Time: 3.81835
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000002025f numSplitK: 1 numBuffers: 0 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000002025f Time: 20.8252
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000204025f numSplitK: 2 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000204025f Time: 10.4449
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000206025f numSplitK: 3 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000206025f Time: 6.99655
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000208025f numSplitK: 4 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000208025f Time: 5.26789
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x00000000020a025f numSplitK: 5 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x00000000020a025f Time: 4.25999
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x00000000020c025f numSplitK: 6 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x00000000020c025f Time: 3.58347
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000210025f numSplitK: 8 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000210025f Time: 5.31732
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000218025f numSplitK: 12 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000218025f Time: 3.65363
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000220025f numSplitK: 16 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000220025f Time: 4.13667
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000230025f numSplitK: 24 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000230025f Time: 3.88359
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000240025f numSplitK: 32 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000240025f Time: 3.65188
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000260025f numSplitK: 48 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000260025f Time: 3.61808
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_128x64_nn_v0 Tactic: 0x000000000280025f numSplitK: 64 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:24] [V] [TRT] Tactic: 0x000000000280025f Time: 3.62013
[12/21/2023-13:05:24] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000002046f numSplitK: 1 numBuffers: 0 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000002046f Time: 41.6057
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000204046f numSplitK: 2 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000204046f Time: 20.8491
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000206046f numSplitK: 3 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000206046f Time: 13.9726
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000208046f numSplitK: 4 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000208046f Time: 10.5114
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x00000000020a046f numSplitK: 5 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x00000000020a046f Time: 8.48559
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x00000000020c046f numSplitK: 6 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x00000000020c046f Time: 7.15132
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000210046f numSplitK: 8 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000210046f Time: 5.43027
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000218046f numSplitK: 12 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000218046f Time: 3.83386
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000220046f numSplitK: 16 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000220046f Time: 5.6535
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000230046f numSplitK: 24 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000230046f Time: 4.24448
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000240046f numSplitK: 32 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000240046f Time: 4.44767
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000260046f numSplitK: 48 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000260046f Time: 4.19416
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_nn_v0 Tactic: 0x000000000280046f numSplitK: 64 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:25] [V] [TRT] Tactic: 0x000000000280046f Time: 4.14552
[12/21/2023-13:05:25] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000002047e numSplitK: 1 numBuffers: 0 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000002047e Time: 41.6054
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000204047e numSplitK: 2 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000204047e Time: 20.85
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000206047e numSplitK: 3 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000206047e Time: 13.9747
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000208047e numSplitK: 4 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000208047e Time: 10.5112
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x00000000020a047e numSplitK: 5 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x00000000020a047e Time: 8.48779
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x00000000020c047e numSplitK: 6 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x00000000020c047e Time: 7.15072
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000210047e numSplitK: 8 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000210047e Time: 5.431
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000218047e numSplitK: 12 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000218047e Time: 3.83444
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000220047e numSplitK: 16 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000220047e Time: 5.65994
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000230047e numSplitK: 24 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000230047e Time: 4.25033
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000240047e numSplitK: 32 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000240047e Time: 4.44972
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000260047e numSplitK: 48 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000260047e Time: 4.19474
[12/21/2023-13:05:26] [V] [TRT] Set Tactic Name: maxwell_hgemm_256x128_tn_v0 Tactic: 0x000000000280047e numSplitK: 64 numBuffers: 1 numKernels: 1
[12/21/2023-13:05:26] [V] [TRT] Tactic: 0x000000000280047e Time: 4.14471
[12/21/2023-13:05:26] [V] [TRT] Fastest Tactic: 0x00000000020c025f Time: 3.58347
[12/21/2023-13:05:26] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskFlattenConvolution)
[12/21/2023-13:05:26] [V] [TRT] CaskFlattenConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:26] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskConvolution)
[12/21/2023-13:05:26] [V] [TRT] CaskConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:26] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CublasConvolution Tactic: 0x0000000000000001
[12/21/2023-13:05:26] [V] [TRT] *************** Autotuning format combination: Half(2432,1:2,1,1) -> Half(256,1,1,1) ***************
[12/21/2023-13:05:26] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskFlattenConvolution)
[12/21/2023-13:05:26] [V] [TRT] CaskFlattenConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:26] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskConvolution)
[12/21/2023-13:05:26] [V] [TRT] CaskConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:26] [V] [TRT] *************** Autotuning format combination: Half(2432,1:2,1,1) -> Half(128,1:2,1,1) ***************
[12/21/2023-13:05:26] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (FusedConvActConvolution)
[12/21/2023-13:05:26] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:26] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CublasConvolution)
[12/21/2023-13:05:26] [V] [TRT] CublasConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:26] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskFlattenConvolution)
[12/21/2023-13:05:26] [V] [TRT] CaskFlattenConvolution has no valid tactics for this config, skipping
[12/21/2023-13:05:26] [V] [TRT] --------------- Timing Runner: /embed/out/out.0/MatMul (CaskConvolution)
[12/21/2023-13:05:26] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_medium_nn_v1 Tactic: 0xc9e631d55d50ffaa
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0xc9e631d55d50ffaa Time: 20.8434
[12/21/2023-13:05:27] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_small_nn_v1 Tactic: 0xc58b636be383c4e6
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0xc58b636be383c4e6 Time: 20.8434
[12/21/2023-13:05:27] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x64_relu_interior_nn_v1 Tactic: 0x714a7bdbc1625db5
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x714a7bdbc1625db5 Time: 10.4293
[12/21/2023-13:05:27] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x32_relu_medium_nn_v1 Tactic: 0x31789dd0dcc2678e
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x31789dd0dcc2678e Time: 10.434
[12/21/2023-13:05:27] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_interior_nn_v1 Tactic: 0xe82e2508fad16cd6
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0xe82e2508fad16cd6 Time: 20.8435
[12/21/2023-13:05:27] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x32_relu_small_nn_v1 Tactic: 0x49d4481f49cb4fcc
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x49d4481f49cb4fcc Time: 10.4334
[12/21/2023-13:05:27] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x64_relu_medium_nn_v1 Tactic: 0x63feec1405ee5560
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x63feec1405ee5560 Time: 10.4296
[12/21/2023-13:05:27] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x32_relu_interior_nn_v1 Tactic: 0x2a8d12e8ad654d04
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x2a8d12e8ad654d04 Time: 10.4341
[12/21/2023-13:05:27] [V] [TRT] /embed/out/out.0/MatMul Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x64_relu_small_nn_v1 Tactic: 0xde90ee8a0a9789b7
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0xde90ee8a0a9789b7 Time: 10.4296
[12/21/2023-13:05:27] [V] [TRT] Fastest Tactic: 0x714a7bdbc1625db5 Time: 10.4293
[12/21/2023-13:05:27] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CaskConvolution Tactic: 0x714a7bdbc1625db5
[12/21/2023-13:05:27] [V] [TRT] =============== Computing costs for 
[12/21/2023-13:05:27] [V] [TRT] *************** Autotuning format combination: Float(256,1,1,1) -> Float(4096,256,1) ***************
[12/21/2023-13:05:27] [V] [TRT] --------------- Timing Runner: reshape_after_/embed/out/out.0/MatMul (Shuffle)
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.023987
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.0883611
[12/21/2023-13:05:27] [V] [TRT] Fastest Tactic: 0x0000000000000000 Time: 0.023987
[12/21/2023-13:05:27] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Shuffle Tactic: 0x0000000000000000
[12/21/2023-13:05:27] [V] [TRT] *************** Autotuning format combination: Float(256,1,256,256) -> Float(1,(* 256 B),B) ***************
[12/21/2023-13:05:27] [V] [TRT] --------------- Timing Runner: reshape_after_/embed/out/out.0/MatMul (Shuffle)
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.0210475
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.0868709
[12/21/2023-13:05:27] [V] [TRT] Fastest Tactic: 0x0000000000000000 Time: 0.0210475
[12/21/2023-13:05:27] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Shuffle Tactic: 0x0000000000000000
[12/21/2023-13:05:27] [V] [TRT] *************** Autotuning format combination: Float(64,1:4,64,64) -> Float(1:4,(* 256 E0),E0) where E0=(CEIL_DIV B 4) where E0=(CEIL_DIV B 4) ***************
[12/21/2023-13:05:27] [V] [TRT] --------------- Timing Runner: reshape_after_/embed/out/out.0/MatMul (Shuffle)
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.0204839
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.111739
[12/21/2023-13:05:27] [V] [TRT] Fastest Tactic: 0x0000000000000000 Time: 0.0204839
[12/21/2023-13:05:27] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Shuffle Tactic: 0x0000000000000000
[12/21/2023-13:05:27] [V] [TRT] *************** Autotuning format combination: Float(8,1:32,1,1) -> Float(4096:32,256,1) ***************
[12/21/2023-13:05:27] [V] [TRT] --------------- Timing Runner: reshape_after_/embed/out/out.0/MatMul (Shuffle)
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.023869
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.110807
[12/21/2023-13:05:27] [V] [TRT] Fastest Tactic: 0x0000000000000000 Time: 0.023869
[12/21/2023-13:05:27] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Shuffle Tactic: 0x0000000000000000
[12/21/2023-13:05:27] [V] [TRT] *************** Autotuning format combination: Half(256,1,1,1) -> Half(4096,256,1) ***************
[12/21/2023-13:05:27] [V] [TRT] --------------- Timing Runner: reshape_after_/embed/out/out.0/MatMul (Shuffle)
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.0188966
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.0662491
[12/21/2023-13:05:27] [V] [TRT] Fastest Tactic: 0x0000000000000000 Time: 0.0188966
[12/21/2023-13:05:27] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Shuffle Tactic: 0x0000000000000000
[12/21/2023-13:05:27] [V] [TRT] *************** Autotuning format combination: Half(128,1:2,1,1) -> Half(4096:2,256,1) ***************
[12/21/2023-13:05:27] [V] [TRT] --------------- Timing Runner: reshape_after_/embed/out/out.0/MatMul (Shuffle)
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.0184594
[12/21/2023-13:05:27] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.112009
[12/21/2023-13:05:28] [V] [TRT] Fastest Tactic: 0x0000000000000000 Time: 0.0184594
[12/21/2023-13:05:28] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Shuffle Tactic: 0x0000000000000000
[12/21/2023-13:05:28] [V] [TRT] =============== Computing costs for 
[12/21/2023-13:05:28] [V] [TRT] *************** Autotuning format combination: Float(256,256,1), Float(4096,256,1) -> Float(4096,256,1) ***************
[12/21/2023-13:05:28] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWiseV2)
[12/21/2023-13:05:28] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.00446871
[12/21/2023-13:05:28] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.00446743
[12/21/2023-13:05:28] [V] [TRT] Tactic: 0x0000000000000002 Time: 0.00444093
[12/21/2023-13:05:28] [V] [TRT] Tactic: 0x0000000000000003 Time: 0.0055956
[12/21/2023-13:05:29] [V] [TRT] Tactic: 0x0000000000000004 Time: 0.00469751
[12/21/2023-13:05:29] [V] [TRT] Tactic: 0x0000000000000005 Time: 0.00455757
[12/21/2023-13:05:29] [V] [TRT] Tactic: 0x0000000000000006 Time: 0.00495653
[12/21/2023-13:05:29] [V] [TRT] Tactic: 0x0000000000000007 Time: 0.00464
[12/21/2023-13:05:29] [V] [TRT] Tactic: 0x0000000000000008 Time: 0.00442819
[12/21/2023-13:05:30] [V] [TRT] Tactic: 0x0000000000000009 Time: 0.00454457
[12/21/2023-13:05:30] [V] [TRT] Tactic: 0x000000000000001c Time: 0.00442223
[12/21/2023-13:05:30] [V] [TRT] Fastest Tactic: 0x000000000000001c Time: 0.00442223
[12/21/2023-13:05:30] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWise)
[12/21/2023-13:05:30] [V] [TRT] PointWise has no valid tactics for this config, skipping
[12/21/2023-13:05:30] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PointWiseV2 Tactic: 0x000000000000001c
[12/21/2023-13:05:30] [V] [TRT] *************** Autotuning format combination: Float(1,256,1), Float(1,(* 256 B),B) -> Float(1,(* 256 B),B) where E0=(* 256 B) ***************
[12/21/2023-13:05:30] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWiseV2)
[12/21/2023-13:05:30] [V] [TRT] PointWiseV2 has no valid tactics for this config, skipping
[12/21/2023-13:05:30] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWise)
[12/21/2023-13:05:30] [V] [TRT] PointWise has no valid tactics for this config, skipping
[12/21/2023-13:05:30] [V] [TRT] *************** Autotuning format combination: Float(1:4,256,1), Float(1:4,(* 256 E0),E0) where E0=(CEIL_DIV B 4) -> Float(1:4,(* 256 E0),E0) where E0=(CEIL_DIV B 4) where E0=(CEIL_DIV B 4) E1=(* 256 E0) ***************
[12/21/2023-13:05:30] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWiseV2)
[12/21/2023-13:05:30] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.00803505
[12/21/2023-13:05:30] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.00899257
[12/21/2023-13:05:30] [V] [TRT] Tactic: 0x0000000000000002 Time: 0.0121596
[12/21/2023-13:05:31] [V] [TRT] Tactic: 0x0000000000000003 Time: 0.0101815
[12/21/2023-13:05:31] [V] [TRT] Tactic: 0x0000000000000004 Time: 0.0136844
[12/21/2023-13:05:31] [V] [TRT] Tactic: 0x0000000000000005 Time: 0.00865398
[12/21/2023-13:05:31] [V] [TRT] Tactic: 0x0000000000000006 Time: 0.0130859
[12/21/2023-13:05:31] [V] [TRT] Tactic: 0x0000000000000007 Time: 0.015717
[12/21/2023-13:05:32] [V] [TRT] Tactic: 0x0000000000000008 Time: 0.0146162
[12/21/2023-13:05:32] [V] [TRT] Tactic: 0x0000000000000009 Time: 0.0105796
[12/21/2023-13:05:32] [V] [TRT] Tactic: 0x000000000000000a Time: 0.0063519
[12/21/2023-13:05:32] [V] [TRT] Tactic: 0x000000000000000b Time: 0.00759964
[12/21/2023-13:05:33] [V] [TRT] Tactic: 0x000000000000000c Time: 0.0073328
[12/21/2023-13:05:33] [V] [TRT] Tactic: 0x000000000000000d Time: 0.00825346
[12/21/2023-13:05:33] [V] [TRT] Tactic: 0x000000000000000e Time: 0.00934314
[12/21/2023-13:05:33] [V] [TRT] Tactic: 0x000000000000000f Time: 0.00864108
[12/21/2023-13:05:33] [V] [TRT] Tactic: 0x0000000000000010 Time: 0.0101842
[12/21/2023-13:05:34] [V] [TRT] Tactic: 0x0000000000000011 Time: 0.0106772
[12/21/2023-13:05:34] [V] [TRT] Tactic: 0x0000000000000012 Time: 0.0118228
[12/21/2023-13:05:34] [V] [TRT] Tactic: 0x0000000000000013 Time: 0.0073936
[12/21/2023-13:05:34] [V] [TRT] Tactic: 0x0000000000000014 Time: 0.00561899
[12/21/2023-13:05:35] [V] [TRT] Tactic: 0x0000000000000015 Time: 0.00662566
[12/21/2023-13:05:35] [V] [TRT] Tactic: 0x0000000000000016 Time: 0.00784986
[12/21/2023-13:05:35] [V] [TRT] Tactic: 0x0000000000000017 Time: 0.0100958
[12/21/2023-13:05:35] [V] [TRT] Tactic: 0x000000000000001c Time: 0.00279007
[12/21/2023-13:05:35] [V] [TRT] Tactic: 0x000000000000001d Time: 0.00294437
[12/21/2023-13:05:36] [V] [TRT] Tactic: 0x000000000000001e Time: 0.00364194
[12/21/2023-13:05:36] [V] [TRT] Fastest Tactic: 0x000000000000001c Time: 0.00279007
[12/21/2023-13:05:36] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWise)
[12/21/2023-13:05:36] [V] [TRT] PointWise has no valid tactics for this config, skipping
[12/21/2023-13:05:36] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PointWiseV2 Tactic: 0x000000000000001c
[12/21/2023-13:05:36] [V] [TRT] *************** Autotuning format combination: Float(256:32,256,1), Float(4096:32,256,1) -> Float(4096:32,256,1) ***************
[12/21/2023-13:05:36] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWiseV2)
[12/21/2023-13:05:36] [V] [TRT] Tactic: 0x0000000000000018 Time: 0.00359657
[12/21/2023-13:05:36] [V] [TRT] Tactic: 0x0000000000000019 Time: 0.0035784
[12/21/2023-13:05:36] [V] [TRT] Tactic: 0x000000000000001a Time: 0.00433191
[12/21/2023-13:05:36] [V] [TRT] Tactic: 0x000000000000001b Time: 0.00660571
[12/21/2023-13:05:37] [V] [TRT] Tactic: 0x000000000000001f Time: 0.00359417
[12/21/2023-13:05:37] [V] [TRT] Fastest Tactic: 0x0000000000000019 Time: 0.0035784
[12/21/2023-13:05:37] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWise)
[12/21/2023-13:05:37] [V] [TRT] PointWise has no valid tactics for this config, skipping
[12/21/2023-13:05:37] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PointWiseV2 Tactic: 0x0000000000000019
[12/21/2023-13:05:37] [V] [TRT] *************** Autotuning format combination: Half(256,256,1), Half(4096,256,1) -> Half(4096,256,1) ***************
[12/21/2023-13:05:37] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWiseV2)
[12/21/2023-13:05:37] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.00358869
[12/21/2023-13:05:37] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.003323
[12/21/2023-13:05:37] [V] [TRT] Tactic: 0x0000000000000002 Time: 0.00340625
[12/21/2023-13:05:37] [V] [TRT] Tactic: 0x0000000000000003 Time: 0.00348691
[12/21/2023-13:05:38] [V] [TRT] Tactic: 0x0000000000000004 Time: 0.00325101
[12/21/2023-13:05:38] [V] [TRT] Tactic: 0x0000000000000005 Time: 0.00328727
[12/21/2023-13:05:38] [V] [TRT] Tactic: 0x0000000000000006 Time: 0.00420276
[12/21/2023-13:05:38] [V] [TRT] Tactic: 0x0000000000000007 Time: 0.00393303
[12/21/2023-13:05:38] [V] [TRT] Tactic: 0x0000000000000008 Time: 0.0036536
[12/21/2023-13:05:39] [V] [TRT] Tactic: 0x0000000000000009 Time: 0.00340561
[12/21/2023-13:05:39] [V] [TRT] Tactic: 0x000000000000001c Time: 0.00347407
[12/21/2023-13:05:39] [V] [TRT] Fastest Tactic: 0x0000000000000004 Time: 0.00325101
[12/21/2023-13:05:39] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWise)
[12/21/2023-13:05:39] [V] [TRT] PointWise has no valid tactics for this config, skipping
[12/21/2023-13:05:39] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PointWiseV2 Tactic: 0x0000000000000004
[12/21/2023-13:05:39] [V] [TRT] *************** Autotuning format combination: Half(256:2,256,1), Half(4096:2,256,1) -> Half(4096:2,256,1) ***************
[12/21/2023-13:05:39] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWiseV2)
[12/21/2023-13:05:39] [V] [TRT] Tactic: 0x0000000000000000 Time: 0.00362263
[12/21/2023-13:05:39] [V] [TRT] Tactic: 0x0000000000000001 Time: 0.00347766
[12/21/2023-13:05:39] [V] [TRT] Tactic: 0x0000000000000002 Time: 0.00354999
[12/21/2023-13:05:40] [V] [TRT] Tactic: 0x0000000000000003 Time: 0.00436502
[12/21/2023-13:05:40] [V] [TRT] Tactic: 0x0000000000000004 Time: 0.00449629
[12/21/2023-13:05:40] [V] [TRT] Tactic: 0x0000000000000005 Time: 0.00453343
[12/21/2023-13:05:40] [V] [TRT] Tactic: 0x0000000000000006 Time: 0.00484434
[12/21/2023-13:05:41] [V] [TRT] Tactic: 0x0000000000000007 Time: 0.00492832
[12/21/2023-13:05:41] [V] [TRT] Tactic: 0x0000000000000008 Time: 0.00517518
[12/21/2023-13:05:41] [V] [TRT] Tactic: 0x0000000000000009 Time: 0.00539716
[12/21/2023-13:05:41] [V] [TRT] Tactic: 0x000000000000000a Time: 0.00381101
[12/21/2023-13:05:41] [V] [TRT] Tactic: 0x000000000000000b Time: 0.00357737
[12/21/2023-13:05:42] [V] [TRT] Tactic: 0x000000000000000c Time: 0.00346057
[12/21/2023-13:05:42] [V] [TRT] Tactic: 0x000000000000000d Time: 0.00333523
[12/21/2023-13:05:42] [V] [TRT] Tactic: 0x000000000000000e Time: 0.00332821
[12/21/2023-13:05:42] [V] [TRT] Tactic: 0x000000000000000f Time: 0.00330026
[12/21/2023-13:05:42] [V] [TRT] Tactic: 0x0000000000000010 Time: 0.00425277
[12/21/2023-13:05:43] [V] [TRT] Tactic: 0x0000000000000011 Time: 0.00402908
[12/21/2023-13:05:43] [V] [TRT] Tactic: 0x0000000000000012 Time: 0.00415948
[12/21/2023-13:05:43] [V] [TRT] Tactic: 0x0000000000000013 Time: 0.00386842
[12/21/2023-13:05:43] [V] [TRT] Tactic: 0x000000000000001c Time: 0.00357726
[12/21/2023-13:05:43] [V] [TRT] Tactic: 0x000000000000001d Time: 0.00377263
[12/21/2023-13:05:43] [V] [TRT] Fastest Tactic: 0x000000000000000f Time: 0.00330026
[12/21/2023-13:05:43] [V] [TRT] --------------- Timing Runner: PWN(/embed/out/out.0/Add, PWN(/embed/pos_enc/Constant_output_0 + (Unnamed Layer* 75) [Shuffle], /embed/pos_enc/Mul)) (PointWise)
[12/21/2023-13:05:43] [V] [TRT] PointWise has no valid tactics for this config, skipping
[12/21/2023-13:05:43] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PointWiseV2 Tactic: 0x000000000000000f
[12/21/2023-13:05:43] [V] [TRT] =============== Computing costs for 
[12/21/2023-13:05:43] [V] [TRT] *************** Autotuning format combination: Float(4096,256,1), Float(1), Float(1) -> Float(4096,256,1) ***************
[12/21/2023-13:05:43] [V] [TRT] --------------- Timing Runner: LayerNormN-0 (PluginV2)
[12/21/2023-13:05:44] [V] [TRT] Deleting timing cache: 845 entries, served 18643 hits since creation.
[12/21/2023-13:05:44] [E] Error[2]: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
[12/21/2023-13:05:44] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[12/21/2023-13:05:44] [E] Engine could not be created from network
[12/21/2023-13:05:44] [E] Building engine failed
[12/21/2023-13:05:44] [E] Failed to create engine from model or file.
[12/21/2023-13:05:44] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8501] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoderV4.onnx --minShapes=chunk_xs:1x67x80,chunk_lens:1,offset:1x1,att_cache:1x12x4x80x128,cnn_cache:1x12x256x7,cache_mask:1x1x80 --optShapes=chunk_xs:16x67x80,chunk_lens:16,offset:16x1,att_cache:16x12x4x80x128,cnn_cache:16x12x256x7,cache_mask:16x1x80 --maxShapes=chunk_xs:32x67x80,chunk_lens:32,offset:32x1,att_cache:32x12x4x80x128,cnn_cache:32x12x256x7,cache_mask:32x1x80 --plugins=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/LayerNorm.so --saveEngine=//mnt/xt/8T/CODES/gitlab-ubt/husky/wenet/runtime/gpu/tensorrt/exp_streaming_trt/encoder_fp16.plan --verbose

I noticed a warning here when running trtexec:

[12/21/2023-12:57:51] [W] [TRT] parsers/onnx/builtin_op_importers.cpp:5225: Attribute epsilon not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.

Does it matter?

wenet-e2e / wenet