search

Tencent / TPAT

TensorRT Plugin Autogen Tool
Apache License 2.0
365 stars 42 forks source link

Could you provide simple tutorial on how to run onnx_to_plugin for simple operator? #35

Closed mhmmdjafarg closed 1 year ago

mhmmdjafarg commented 1 year ago

Hi, thank you for your great work. I just wonder how to run onnx_to_plugin on Tile operator. I know it is supported by TPAT 1.0. I have tried python3 onnx_to_plugin.py -i model/pfe_baseline32000.onnx -o model/pfe_baseline_tpat.onnx -t Tile python3 onnx_to_plugin.py -i model/pfe_baseline32000.onnx -o model/pfe_baseline_tpat.onnx -n Tile_16 -dynamic=true -min=1 -max=256 -opt=128 But it returns

Couldn't find reusable plugin for node Tile_16

  7: tvm::relay::StorageAllocaBaseVisitor::DeviceAwareVisitExpr_(tvm::relay::FunctionNode const*)                                  [0/60]
  6: tvm::relay::StorageAllocaBaseVisitor::GetToken(tvm::RelayExpr const&)
  5: tvm::relay::ExprVisitor::VisitExpr(tvm::RelayExpr const&)
  4: tvm::relay::transform::DeviceAwareExprVisitor::VisitExpr_(tvm::relay::CallNode const*)
  3: tvm::relay::StorageAllocator::DeviceAwareVisitExpr_(tvm::relay::CallNode const*)
  2: tvm::relay::StorageAllocaBaseVisitor::CreateToken(tvm::RelayExprNode const*, bool)
  1: tvm::relay::StorageAllocator::CreateTokenOnDevice(tvm::RelayExprNode const*, DLDeviceType, bool)
  0: tvm::relay::StorageAllocator::GetMemorySize(tvm::relay::StorageToken*)
  File "/workspace/TPAT/3rdparty/blazerml-tvm/src/relay/backend/graph_plan_memory.cc", line 408
TVMError:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (pval != nullptr) is false: Cannot allocate memory symbolic tensor shape [?, ?, ?]

Thank you

wenqf11 commented 1 year ago

@mhmmdjafarg you mean dynamic=false is okay but when dynamic=true, it fails? Could you provide your onnx model?

mhmmdjafarg commented 1 year ago

No, both are returns error. I just wonder what is the correct way to do it? Maybe like the one you did on Resize operator Here's the onnx file https://drive.google.com/file/d/1WTJe7GNknIqt6cOi9C-izY9D1bKqvDjk/view?usp=sharing

wenqf11 commented 1 year ago

@mhmmdjafarg Currently, we only support Tile operator with specific shape, can not support symbolic shape. You should fix your Tile shape except the first dimension(batch_size dimension).

mhmmdjafarg commented 1 year ago

Thank you for your reply, now I understand. Would you happen to have any idea how to make it fixed? My input tensor should be with shape (32000,1,32) and the tile is 1D with value (1,20,1)

I found the solution using onnx.helper, adding the value_info on the graph. I'll get back to you later. thanks