search

microsoft / nnfusion

A flexible and efficient deep neural network (DNN) compiler that generates high-performance executable from a DNN model description.
MIT License
948 stars 158 forks source link

[BUG] Compile lstm.onnx failed. #441

Open LeiWang1999 opened 2 years ago

LeiWang1999 commented 2 years ago

🐛 Bug

I have a converted lstm model located in /data1/v-leiwang3/benchmark/nnfusion_models/lstm.float32.onnx, which can be inferenced by onnxruntime, but I got a compiled failed issue with nnfusion.

To Reproduce

nnfusion /workspace/v-leiwang3/benchmark/nnfusion_models/lstm.float32.onnx -f onnx -p batch_size:1;seq_length:512 -fwarmup_step=5 -frun_step=100
[WARNING] 2022-06-05T05:12:58z src/contrib/custom_op/custom_op.h 27 $NNFUSION_HOME was not set, use /root/.nnfusion.
[WARNING] 2022-06-05T05:12:58z src/contrib/custom_op/custom_op.h 27 $NNFUSION_HOME was not set, use /root/.nnfusion.

============================================================================
---- Processing '/workspace/v-leiwang3/benchmark/nnfusion_models/lstm.float32.onnx'
============================================================================
[INFO] 2022-06-05T05:12:58z src/nnfusion/frontend/onnx_import/onnx.cpp 54   Optimizing ONNX Graph with External Tool (models/pytorch2onnx/ort_run_frozen.py)
[WARNING] 2022-06-05T05:12:58z src/nnfusion/common/util.cpp 47  $NNFUSION_HOME was not set, use /root/.nnfusion.
ONNX model check passed!
Importing ONNX model into ONNX Runtime...
Execution Providers: ['CUDAExecutionProvider', 'CPUExecutionProvider']
output
[-0.00914649  0.01211212  0.03566384  0.02083438  0.02440321  0.02316982
  0.01730602  0.00079239 -0.01367441 -0.01027629] ...(size= 100 end with -0.010356168 )
[INFO] 2022-06-05T05:13:02z src/nnfusion/frontend/onnx_import/onnx.cpp 40   Import ONNX Graph Size: [256593293]
[INFO] 2022-06-05T05:13:02z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 317    Converting Onnx Graph
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Transpose_6
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Transpose_6, output 45, shape Shape{512, 1, 256}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: LSTM_15
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node LSTM_15, output 111, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_16
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_16, output 114, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: LSTM_25
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node LSTM_25, output 179, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_26
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_26, output 182, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: LSTM_35
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node LSTM_35, output 247, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_36
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_36, output 250, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: LSTM_45
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node LSTM_45, output 315, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_46
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_46, output 318, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: LSTM_55
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node LSTM_55, output 383, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_56
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_56, output 386, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: LSTM_65
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node LSTM_65, output 451, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_66
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_66, output 454, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: LSTM_75
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node LSTM_75, output 519, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_76
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_76, output 522, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: LSTM_85
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node LSTM_85, output 587, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_86
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_86, output 590, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Slice_91
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Slice_91, output Slice_91, shape Shape{1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Transpose
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Transpose, output 595, shape Shape{1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Squeeze_92
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Squeeze_92, output 596, shape Shape{1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519    convert node: Gemm_93
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538    node Gemm_93, output output, shape Shape{1, 100}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 514    convert graph done
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/engine.cpp 36   Graph Passes count:27
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/engine.cpp 37   Interpreter Passes count:6
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/pass/graph/batchnorm_inference_folding_pass.cpp 889 batchnorm inference folding Pass starts up for Graph: Graph_1
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/pass/graph/batchnorm_inference_folding_pass.cpp 901 batchnorm inference folding Pass ends for Graph: Graph_1
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/cache/manager.cpp 52    Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/cache/manager.cpp 52    Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:input(op type: Parameter, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:LSTM_15(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:LSTM_25(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:LSTM_35(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:LSTM_45(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:LSTM_55(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:LSTM_65(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:LSTM_75(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270    No valid kernel found:LSTM_85(op type: Lstm, dev type: CUDA_GPU)
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/cache/manager.cpp 52    Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/cache/manager.cpp 52    Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:input
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator Reshape_67 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 45 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:LSTM_15
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 114 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:LSTM_25
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 182 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:LSTM_35
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 250 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:LSTM_45
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 318 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:LSTM_55
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 386 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:LSTM_65
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 454 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:LSTM_75
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 522 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153 Kernel should be emitted before this pass:LSTM_85
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 590 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 595 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator 596 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator Dot_66 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184    Operator Result_69 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node _linear.bias
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node Reshape_67
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node _linear.weight
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 586
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 582
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 759
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 758
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 757
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 518
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 514
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 739
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 738
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 737
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 450
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 446
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 719
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 718
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 717
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 382
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 378
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 699
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 698
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 697
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 314
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 310
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 679
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 678
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 677
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 246
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 242
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 659
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 658
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 657
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 178
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 174
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 639
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 638
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 637
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 110
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 106
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 619
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 618
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 617
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node input
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 45
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node LSTM_15
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 114
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node LSTM_25
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 182
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node LSTM_35
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 250
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node LSTM_45
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 318
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node LSTM_55
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 386
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node LSTM_65
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 454
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node LSTM_75
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 522
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node LSTM_85
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 590
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node Slice_91
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 595
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node 596
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node Dot_66
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node Add_68
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100   processed node Result_69
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 213   assign thread info-------------------------------
[ERROR] 2022-06-05T05:13:05z src/nnfusion/util/errors.hpp 169   Check failed: '(*gnode)["Kernel_Selection_Result"].is_valid()' at /workspace/v-leiwang3/nnfusion/src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp:926:
Kernel should be selected before this pass:Lstm kernel name : LSTM_15
terminate called after throwing an instance of 'nnfusion::errors::CheckError'
  what():  Check failed: '(*gnode)["Kernel_Selection_Result"].is_valid()' at /workspace/v-leiwang3/nnfusion/src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp:926:
Kernel should be selected before this pass:Lstm kernel name : LSTM_15

some additional debug logs has been added by myself to find the problem, It seems that something stucked at lstm node, any suggestion?

LeiWang1999 commented 2 years ago

I think the problem is because nnfusion has no Lstm Op Cuda_GPU DeviceType kernel, because there is no lstm.cpp under folder core/kernels/cuda_gpu/kernels.