search

modelscope / FunASR

A Fundamental End-to-End Speech Recognition Toolkit and Open Source SOTA Pretrained Models, Supporting Speech Recognition, Voice Activity Detection, Text Post-processing etc.
https://www.funasr.com
Other
7.08k stars 755 forks source link

sensevoice convert onnx to triton fail #2226

Open didadida-r opened 3 days ago

didadida-r commented 3 days ago

Notice: In order to resolve issues more efficiently, please raise issue following the template. （注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节）

🐛 Bug

To Reproduce

Steps to reproduce the behavior (always include the command you ran):

  1. Run cmd '....' 
    TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --onnx=./model_repo_sense_voice_small/encoder/1/model.onnx --saveEngine=model.plan --minShapes=speech:1x8x560,speech_lengths:1,language:1,textnorm:1 --optShapes=speech:4x128x560,speech_lengths:4,language:4,textnorm:4 --maxShapes=speech:16x512x560,speech_lengths:16,language:16,textnorm:16
  2. See error
[11/22/2024-07:40:48] [I] === Model Options ===
[11/22/2024-07:40:48] [I] Format: ONNX
[11/22/2024-07:40:48] [I] Model: ./model_repo_sense_voice_small/encoder/1/model.onnx
[11/22/2024-07:40:48] [I] Output:
[11/22/2024-07:40:48] [I] === Build Options ===
[11/22/2024-07:40:48] [I] Max batch: explicit batch
[11/22/2024-07:40:48] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[11/22/2024-07:40:48] [I] minTiming: 1
[11/22/2024-07:40:48] [I] avgTiming: 8
[11/22/2024-07:40:48] [I] Precision: FP32
[11/22/2024-07:40:48] [I] LayerPrecisions:
[11/22/2024-07:40:48] [I] Calibration:
[11/22/2024-07:40:48] [I] Refit: Disabled
[11/22/2024-07:40:48] [I] Sparsity: Disabled
[11/22/2024-07:40:48] [I] Safe mode: Disabled
[11/22/2024-07:40:48] [I] DirectIO mode: Disabled
[11/22/2024-07:40:48] [I] Restricted mode: Disabled
[11/22/2024-07:40:48] [I] Build only: Disabled
[11/22/2024-07:40:48] [I] Save engine: model.plan
[11/22/2024-07:40:48] [I] Load engine:
[11/22/2024-07:40:48] [I] Profiling verbosity: 0
[11/22/2024-07:40:48] [I] Tactic sources: Using default tactic sources
[11/22/2024-07:40:48] [I] timingCacheMode: local
[11/22/2024-07:40:48] [I] timingCacheFile:
[11/22/2024-07:40:48] [I] Heuristic: Disabled
[11/22/2024-07:40:48] [I] Preview Features: Use default preview flags.
[11/22/2024-07:40:48] [I] Input(s)s format: fp32:CHW
[11/22/2024-07:40:48] [I] Output(s)s format: fp32:CHW
[11/22/2024-07:40:48] [I] Input build shape: speech=1x8x560+4x128x560+16x512x560
[11/22/2024-07:40:48] [I] Input build shape: speech_lengths=1+4+16
[11/22/2024-07:40:48] [I] Input build shape: textnorm=1+4+16
[11/22/2024-07:40:48] [I] Input build shape: language=1+4+16
[11/22/2024-07:40:48] [I] Input calibration shapes: model
[11/22/2024-07:40:48] [I] === System Options ===
[11/22/2024-07:40:48] [I] Device: 0
[11/22/2024-07:40:48] [I] DLACore:
[11/22/2024-07:40:48] [I] Plugins:
[11/22/2024-07:40:48] [I] === Inference Options ===
[11/22/2024-07:40:48] [I] Batch: Explicit
[11/22/2024-07:40:48] [I] Input inference shape: language=4
[11/22/2024-07:40:48] [I] Input inference shape: textnorm=4
[11/22/2024-07:40:48] [I] Input inference shape: speech_lengths=4
[11/22/2024-07:40:48] [I] Input inference shape: speech=4x128x560
[11/22/2024-07:40:48] [I] Iterations: 10
[11/22/2024-07:40:48] [I] Duration: 3s (+ 200ms warm up)
[11/22/2024-07:40:48] [I] Sleep time: 0ms
[11/22/2024-07:40:48] [I] Idle time: 0ms
[11/22/2024-07:40:48] [I] Streams: 1
[11/22/2024-07:40:48] [I] ExposeDMA: Disabled
[11/22/2024-07:40:48] [I] Data transfers: Enabled
[11/22/2024-07:40:48] [I] Spin-wait: Disabled
[11/22/2024-07:40:48] [I] Multithreading: Disabled
[11/22/2024-07:40:48] [I] CUDA Graph: Disabled
[11/22/2024-07:40:48] [I] Separate profiling: Disabled
[11/22/2024-07:40:48] [I] Time Deserialize: Disabled
[11/22/2024-07:40:48] [I] Time Refit: Disabled
[11/22/2024-07:40:48] [I] NVTX verbosity: 0
[11/22/2024-07:40:48] [I] Persistent Cache Ratio: 0
[11/22/2024-07:40:48] [I] Inputs:
[11/22/2024-07:40:48] [I] === Reporting Options ===
[11/22/2024-07:40:48] [I] Verbose: Disabled
[11/22/2024-07:40:48] [I] Averages: 10 inferences
[11/22/2024-07:40:48] [I] Percentiles: 90,95,99
[11/22/2024-07:40:48] [I] Dump refittable layers:Disabled
[11/22/2024-07:40:48] [I] Dump output: Disabled
[11/22/2024-07:40:48] [I] Profile: Disabled
[11/22/2024-07:40:48] [I] Export timing to JSON file:
[11/22/2024-07:40:48] [I] Export output to JSON file:
[11/22/2024-07:40:48] [I] Export profile to JSON file:
[11/22/2024-07:40:48] [I]
[11/22/2024-07:40:55] [I] === Device Information ===
[11/22/2024-07:40:55] [I] Selected Device: NVIDIA GeForce RTX 2080 Ti
[11/22/2024-07:40:55] [I] Compute Capability: 7.5
[11/22/2024-07:40:55] [I] SMs: 68
[11/22/2024-07:40:55] [I] Compute Clock Rate: 1.545 GHz
[11/22/2024-07:40:55] [I] Device Global Memory: 11011 MiB
[11/22/2024-07:40:55] [I] Shared Memory per SM: 64 KiB
[11/22/2024-07:40:55] [I] Memory Bus Width: 352 bits (ECC disabled)
[11/22/2024-07:40:55] [I] Memory Clock Rate: 7 GHz
[11/22/2024-07:40:55] [I]
[11/22/2024-07:40:55] [I] TensorRT version: 8.5.2
[11/22/2024-07:40:55] [I] [TRT] [MemUsageChange] Init CUDA: CPU +14, GPU +0, now: CPU 27, GPU 157 (MiB)
[11/22/2024-07:40:58] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +342, GPU +76, now: CPU 423, GPU 233 (MiB)
[11/22/2024-07:40:58] [I] Start parsing network model
[11/22/2024-07:40:59] [I] [TRT] ----------------------------------------------------------------
[11/22/2024-07:40:59] [I] [TRT] Input filename:   ./model_repo_sense_voice_small/encoder/1/model.onnx
[11/22/2024-07:40:59] [I] [TRT] ONNX IR version:  0.0.8
[11/22/2024-07:40:59] [I] [TRT] Opset version:    18
[11/22/2024-07:40:59] [I] [TRT] Producer name:    pytorch
[11/22/2024-07:40:59] [I] [TRT] Producer version: 2.3.0
[11/22/2024-07:40:59] [I] [TRT] Domain:
[11/22/2024-07:40:59] [I] [TRT] Model version:    0
[11/22/2024-07:40:59] [I] [TRT] Doc string:
[11/22/2024-07:40:59] [I] [TRT] ----------------------------------------------------------------
[11/22/2024-07:41:00] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[11/22/2024-07:41:00] [I] [TRT] No importer registered for op: LayerNormalization. Attempting to import as plugin.
[11/22/2024-07:41:00] [I] [TRT] Searching for plugin: LayerNormalization, plugin_version: 1, plugin_namespace:
[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:726: While parsing node number 101 [LayerNormalization -> "/encoder/encoders0.0/norm1/LayerNormalization_output_0"]:
[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:727: --- Begin node ---
[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:728: input: "/encoder/encoders0.0/norm1/Cast_output_0"
input: "onnx::LayerNormalization_10442"
input: "onnx::LayerNormalization_10443"
output: "/encoder/encoders0.0/norm1/LayerNormalization_output_0"
name: "/encoder/encoders0.0/norm1/LayerNormalization"
op_type: "LayerNormalization"
attribute {
  name: "axis"
  i: -1
  type: INT
}
attribute {
  name: "epsilon"
  f: 1e-05
  type: FLOAT
}

[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:729: --- End node ---
[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:731: ERROR: parsers/onnx/builtin_op_importers.cpp:5429 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[11/22/2024-07:41:00] [E] Failed to parse onnx file
[11/22/2024-07:41:00] [I] Finish parsing network model
[11/22/2024-07:41:00] [E] Parsing model failed
[11/22/2024-07:41:00] [E] Failed to create engine from model or file.
[11/22/2024-07:41:00] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --onnx=./model_repo_sense_voice_small/encoder/1/model.onnx --saveEngine=model.plan --minShapes=speech:1x8x560,speech_lengths:1,language:1,textnorm:1 --optShapes=speech:4x128x560,speech_lengths:4,language:4,textnorm:4 --maxShapes=speech:16x512x560,speech_lengths:16,language:16,textnorm:16
&&&& RUNNING TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --loadEngine=model.plan --noDataTransfers --verbose --shapes=speech:4x65x560,speech_lengths:4,language:4,textnorm:4
[11/22/2024-07:41:01] [I] === Model Options ===
[11/22/2024-07:41:01] [I] Format: *
[11/22/2024-07:41:01] [I] Model:
[11/22/2024-07:41:01] [I] Output:
[11/22/2024-07:41:01] [I] === Build Options ===
[11/22/2024-07:41:01] [I] Max batch: explicit batch
[11/22/2024-07:41:01] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[11/22/2024-07:41:01] [I] minTiming: 1
[11/22/2024-07:41:01] [I] avgTiming: 8
[11/22/2024-07:41:01] [I] Precision: FP32
[11/22/2024-07:41:01] [I] LayerPrecisions:
[11/22/2024-07:41:01] [I] Calibration:
[11/22/2024-07:41:01] [I] Refit: Disabled
[11/22/2024-07:41:01] [I] Sparsity: Disabled
[11/22/2024-07:41:01] [I] Safe mode: Disabled
[11/22/2024-07:41:01] [I] DirectIO mode: Disabled
[11/22/2024-07:41:01] [I] Restricted mode: Disabled
[11/22/2024-07:41:01] [I] Build only: Disabled
[11/22/2024-07:41:01] [I] Save engine:
[11/22/2024-07:41:01] [I] Load engine: model.plan
[11/22/2024-07:41:01] [I] Profiling verbosity: 0
[11/22/2024-07:41:01] [I] Tactic sources: Using default tactic sources
[11/22/2024-07:41:01] [I] timingCacheMode: local
[11/22/2024-07:41:01] [I] timingCacheFile:
[11/22/2024-07:41:01] [I] Heuristic: Disabled
[11/22/2024-07:41:01] [I] Preview Features: Use default preview flags.
[11/22/2024-07:41:01] [I] Input(s)s format: fp32:CHW
[11/22/2024-07:41:01] [I] Output(s)s format: fp32:CHW
[11/22/2024-07:41:01] [I] Input build shape: speech=4x65x560+4x65x560+4x65x560
[11/22/2024-07:41:01] [I] Input build shape: speech_lengths=4+4+4
[11/22/2024-07:41:01] [I] Input build shape: textnorm=4+4+4
[11/22/2024-07:41:01] [I] Input build shape: language=4+4+4
[11/22/2024-07:41:01] [I] Input calibration shapes: model
[11/22/2024-07:41:01] [I] === System Options ===
[11/22/2024-07:41:01] [I] Device: 0
[11/22/2024-07:41:01] [I] DLACore:
[11/22/2024-07:41:01] [I] Plugins:
[11/22/2024-07:41:01] [I] === Inference Options ===
[11/22/2024-07:41:01] [I] Batch: Explicit
[11/22/2024-07:41:01] [I] Input inference shape: language=4
[11/22/2024-07:41:01] [I] Input inference shape: textnorm=4
[11/22/2024-07:41:01] [I] Input inference shape: speech_lengths=4
[11/22/2024-07:41:01] [I] Input inference shape: speech=4x65x560
[11/22/2024-07:41:01] [I] Iterations: 10
[11/22/2024-07:41:01] [I] Duration: 3s (+ 200ms warm up)
[11/22/2024-07:41:01] [I] Sleep time: 0ms
[11/22/2024-07:41:01] [I] Idle time: 0ms
[11/22/2024-07:41:01] [I] Streams: 1
[11/22/2024-07:41:01] [I] ExposeDMA: Disabled
[11/22/2024-07:41:01] [I] Data transfers: Disabled
[11/22/2024-07:41:01] [I] Spin-wait: Disabled
[11/22/2024-07:41:01] [I] Multithreading: Disabled
[11/22/2024-07:41:01] [I] CUDA Graph: Disabled
[11/22/2024-07:41:01] [I] Separate profiling: Disabled
[11/22/2024-07:41:01] [I] Time Deserialize: Disabled
[11/22/2024-07:41:01] [I] Time Refit: Disabled
[11/22/2024-07:41:01] [I] NVTX verbosity: 0
[11/22/2024-07:41:01] [I] Persistent Cache Ratio: 0
[11/22/2024-07:41:01] [I] Inputs:
[11/22/2024-07:41:01] [I] === Reporting Options ===
[11/22/2024-07:41:01] [I] Verbose: Enabled
[11/22/2024-07:41:01] [I] Averages: 10 inferences
[11/22/2024-07:41:01] [I] Percentiles: 90,95,99
[11/22/2024-07:41:01] [I] Dump refittable layers:Disabled
[11/22/2024-07:41:01] [I] Dump output: Disabled
[11/22/2024-07:41:01] [I] Profile: Disabled
[11/22/2024-07:41:01] [I] Export timing to JSON file:
[11/22/2024-07:41:01] [I] Export output to JSON file:
[11/22/2024-07:41:01] [I] Export profile to JSON file:
[11/22/2024-07:41:01] [I]
[11/22/2024-07:41:08] [I] === Device Information ===
[11/22/2024-07:41:08] [I] Selected Device: NVIDIA GeForce RTX 2080 Ti
[11/22/2024-07:41:08] [I] Compute Capability: 7.5
[11/22/2024-07:41:08] [I] SMs: 68
[11/22/2024-07:41:08] [I] Compute Clock Rate: 1.545 GHz
[11/22/2024-07:41:08] [I] Device Global Memory: 11011 MiB
[11/22/2024-07:41:08] [I] Shared Memory per SM: 64 KiB
[11/22/2024-07:41:08] [I] Memory Bus Width: 352 bits (ECC disabled)
[11/22/2024-07:41:08] [I] Memory Clock Rate: 7 GHz
[11/22/2024-07:41:08] [I]
[11/22/2024-07:41:08] [I] TensorRT version: 8.5.2
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::fMHA_V2 version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::fMHCA version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::GroupNorm version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::LayerNorm version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Proposal version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ProposalDynamic version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::SeqLen2Spatial version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::SplitGeLU version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Split version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[11/22/2024-07:41:08] [E] Error opening engine file: model.plan
[11/22/2024-07:41:08] [E] Failed to create engine from model or file.
[11/22/2024-07:41:08] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --loadEngine=model.plan --noDataTransfers --verbose --shapes=speech:4x65x560,speech_lengths:4,language:4,textnorm:4

Code sample

Expected behavior

Environment

Additional context