search

neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
2.98k stars 172 forks source link

Custom ONNX with LSTM and dynamic batch size #408

Closed Grigor355 closed 2 years ago

Grigor355 commented 2 years ago

I am trying to compile an LSTM based ONNX model, but the kernel dies. It works with CNN based ONNX. Also, it is possible to have a model with dynamic batch size? Ubuntu - 18.04 Python - 3.8 ONNX - 1.9.0 deepsparse - 0.12.1

mgoin commented 2 years ago

Hi @Grigor355, deepsparse hasn't been optimized for LSTM-based models so there is likely to be an issue here. Would you be able to share the code and ONNX file you used to run it?

We don't allow for dynamic batch size at the moment, it is something we plan on addressing but is not ready for performance.