search

Media-Smart / vedastr

A scene text recognition toolbox based on PyTorch
Apache License 2.0
535 stars 100 forks source link

RuntimeError: Exporting the operator grid_sampler to ONNX opset version 9 is not supported #85

Open choozhenbo opened 2 years ago

choozhenbo commented 2 years ago

I would like to convert the TPS-resnet-bilstm-ctc model in ONNX format, but shows the error of "Exporting the operator grid_sampler to ONNX opset version 9 is not supported" This is the command and error messages: python tools/torch2onnx.py configs/tps_resnet_bilstm_ctc.py /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/vedastr/ckpt/tps_resnet_bilstm_ctc.pth /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/tps_resnet_bilstm_ctc.onnx 2022-06-18 15:54:46,461 - INFO - Use GPU 0 2022-06-18 15:54:46,461 - INFO - Set cudnn deterministic False 2022-06-18 15:54:46,461 - INFO - Set cudnn benchmark True 2022-06-18 15:54:46,461 - INFO - Set seed 1111 2022-06-18 15:54:46,461 - INFO - Build model 2022-06-18 15:54:46,786 - INFO - GResNet init weights 2022-06-18 15:54:46,982 - INFO - CTCHead init weights 2022-06-18 15:54:48,754 - INFO - Load checkpoint from /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/vedastr/ckpt/tps_resnet_bilstm_ctc.pth Convert to Onnx with constant input shape 3,32,100 and opset version 9 Traceback (most recent call last): File "tools/torch2onnx.py", line 92, in main() File "tools/torch2onnx.py", line 83, in main do_constant_folding=args.do_constant_folding, File "/home/tham/vedastr/tools/volksdep/converters/torch2onnx.py", line 62, in torch2onnx dynamic_axes=dynamic_axes) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/init.py", line 320, in export custom_opsets, enable_onnx_checker, use_external_data_format) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 111, in export custom_opsets=custom_opsets, use_external_data_format=use_external_data_format) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 729, in _export dynamic_axes=dynamic_axes) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 501, in _model_to_graph module=module) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 216, in _optimize_graph graph = torch._C._jit_pass_onnx(graph, operator_export_type) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/init.py", line 373, in _run_symbolic_function return utils._run_symbolic_function(*args, **kwargs) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 1028, in _run_symbolic_function symbolic_fn = _find_symbolic_in_registry(domain, op_name, opset_version, operator_export_type) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 982, in _find_symbolic_in_registry return sym_registry.get_registered_op(op_name, domain, opset_version) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/symbolic_registry.py", line 125, in get_registered_op raise RuntimeError(msg) RuntimeError: Exporting the operator grid_sampler to ONNX opset version 9 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub

choozhenbo commented 2 years ago

@hxcai do you know where torch.nn.functional.grid_sample is assigned as TPS is included in the model?

choozhenbo commented 2 years ago

@hxcai I found the supported opset version from https://github.com/onnx/onnx/blob/main/docs/Operators.md . But the python tools/torch2onnx.py configs/tps_resnet_bilstm_ctc.py /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/vedastr/ckpt/tps_resnet_bilstm_ctc.pth /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/tps_resnet_bilstm_ctc.onnx --opset_version=16 command shows the error: Convert to Onnx with constant input shape 3,32,100 and opset version 16 Traceback (most recent call last): File "tools/torch2onnx.py", line 92, in main() File "tools/torch2onnx.py", line 83, in main do_constant_folding=args.do_constant_folding, File "/home/tham/vedastr/tools/volksdep/converters/torch2onnx.py", line 62, in torch2onnx dynamic_axes=dynamic_axes) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/init.py", line 320, in export custom_opsets, enable_onnx_checker, use_external_data_format) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 111, in export custom_opsets=custom_opsets, use_external_data_format=use_external_data_format) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 707, in _export _set_opset_version(opset_version) File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 849, in _set_opset_version raise ValueError("Unsupported ONNX opset version: " + str(opset_version)) ValueError: Unsupported ONNX opset version: 16

hxcai commented 2 years ago

@choozhenbo The onnx opset version 16 is not supported by pytorch yet, if you want to support grid_sample, you should implement the operation by yourself.

choozhenbo commented 2 years ago

@hxcai do you have clue how to implement the operation? I tried to trace the location for the implementation of the operation but ended up I have no clue.

hxcai commented 2 years ago

@choozhenbo you can go to this issue to find the solution. You need to add onnx and trt plugin.