search

open-mmlab / mmocr

OpenMMLab Text Detection, Recognition and Understanding Toolbox
https://mmocr.readthedocs.io/en/dev-1.x/
Apache License 2.0
4.3k stars 746 forks source link

How to convert SDMG-R Pytorch model to ONNX? #487

Open anuj-rathore opened 3 years ago

anuj-rathore commented 3 years ago

Deployment for Key Information Extraction

gaotongxiao commented 3 years ago

We don't have a plan for this currently. Ping @AllentDan to see if the deployment team will take this into their future plan. You are also welcome to give it a try and share your progress with our community, if any. :)

anuj-rathore commented 3 years ago

@AllentDan, are you guys looking into it?

AllentDan commented 3 years ago

Hi, @anuj-rathore. SDMG-R is not in our short-term plan. But just like @gaotongxiao said, we welcome that you can give a try on that. Regards.

amitbcp commented 2 years ago

@AllentDan can you provide some pointers on how to start and go about it for SDMGR model ?

Because the framework is building from multiple hooks, I was kinda stuck from where to convert. Some input could help me to create a PR for this

AllentDan commented 2 years ago

Hi, @amitbcp. We have example codes about how to deploy some detection and recognition models in mmocr. Dive into tools/deployment/pytorch2onnx.py and add some codes to make codes execute to torch.onnx.export SDMGR model. Here we probably encounter some unexpected errors. We may use dichotomy to locate the codes that arose errors. To this end, we can use if torch.onnx.is_in_onnx_export() to control whether to run codes that we want to replace with the original codes.

amitbcp commented 2 years ago

Hi @AllentDan !

can you provide some more pointers on how to go forward with the conversion.. even the ONNX inference for SDMGR. Or how you guys do it as a process and I can take it and try to debug it from there?

It will help me move faster and SDMGR will help a lot for community. Looking for your help and guidance

gouttham commented 2 years ago

+1

AllentDan commented 2 years ago

Hi @AllentDan !

can you provide some more pointers on how to go forward with the conversion.. even the ONNX inference for SDMGR. Or how you guys do it as a process and I can take it and try to debug it from there?

It will help me move faster and SDMGR will help a lot for community. Looking for your help and guidance

Let's see, the KIE model needs image tensor, bbox, and text. As onnx does not support string variable input, the text has to be converted to index value and the bbox can be converted to a tensor. You can pass a tuple of these three tensors I guess to torch.onn.export and change the codes inside the KIE model to make it exportable.

That's all I can tell you in detail by now I guess. Best wishes.

amitbcp commented 2 years ago

Thanks @AllentDan , will try it out. Thanks

gouttham commented 2 years ago

Hi @AllentDan I was able to convert the mode into the .onnx file, but I faced an issue during loading this model in onnxruntime Screenshot 2021-10-22 at 11 21 24 AM

on investigating further, I found that the below line in SDMGRHead.py is causing trouble

nodes[valid] = rnn_nodes[valid].gather(1, (all_nums[valid] - 1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, rnn_nodes.size(-1))).squeeze(1)

any suggestions on how to resolve this? can you suggest any alternate snippet ? The onnx model checker has disregarded this, I will raise an issue over there too. Thanks.

gouttham commented 2 years ago

Hi @AllentDan I was able to convert the mode into the .onnx file, but I faced an issue during loading this model in onnxruntime Screenshot 2021-10-22 at 11 21 24 AM

on investigating further, I found that the below line in SDMGRHead.py is causing trouble

nodes[valid] = rnn_nodes[valid].gather(1, (all_nums[valid] - 1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, rnn_nodes.size(-1))).squeeze(1)

any suggestions on how to resolve this? can you suggest any alternate snippet ? The onnx model checker has disregarded this, I will raise an issue over there too. Thanks.

Actually I modified valid = all_nums > 0 as valid = torch.where(all_nums>0)[0] and was able to convert the model into onnx. Thanks

AllentDan commented 2 years ago

Hi, @gouttham. Good job. The line nodes[valid] = rnn_nodes[valid].gather(1, (all_nums[valid] - 1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, rnn_nodes.size(-1))).squeeze(1) is too long to analyse codes. Maybe we can split it to several short lines. BTW, you can pull a request to mmocr and tag a WIP so that we can debug together.

gouttham commented 2 years ago

Hi, @gouttham. Good job. The line nodes[valid] = rnn_nodes[valid].gather(1, (all_nums[valid] - 1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, rnn_nodes.size(-1))).squeeze(1) is too long to analyse codes. Maybe we can split it to several shot lines. BTW, you can pull a request to mmocr and tag a WIP so that we can debug together.

Sure I will create a pr. The actual problem is onnx does not support boolean based indexing ie, valid = all_nums > 0. The issue has been resolved by modifying valid = all_nums > 0 to valid = torch.where(all_nums>0)[0], So we need not worry about the above long line :)

Thanks for the reply. Nice repo :)

syan-dev commented 2 years ago

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

gouttham commented 2 years ago

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Sorry for late reply tofuai. Give me few days, i will raise a PR. Thanks.

xinsuinizhuan commented 2 years ago

how to convert the onnx? I conver the

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Sorry for late reply tofuai. Give me few days, i will raise a PR. Thanks.

could give the expoterd the onnx download link?

lyc728 commented 2 years ago

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

lyc728 commented 2 years ago

Hi @ amitbcp, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

skywalkerfmc commented 2 years ago

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Sorry for late reply tofuai. Give me few days, i will raise a PR. Thanks.

Which pytorch version are you using for convert to onnx? When I set "do_constant_folding=True" in the process of convert to onnx, I got a segmentation fault in pytorch1.7.1.

gouttham commented 2 years ago

Hi @skywalkerfmc , Kindly find below with the version i am using. torch==1.10.0 onnx==1.10.1 onnxruntime==1.6.0

Bourne-M commented 2 years ago

我参考mmocr，写了一个简单版本的，转onnx的时候有些问题，有人能看看吗？谢谢 https://github.com/Bourne-M/SDMG_R

gaotongxiao commented 2 years ago

@Bourne-M Could you elaborate on the issue encountered and the steps to reproduce?

Bourne-M commented 2 years ago

@Bourne-M Could you elaborate on the issue encountered and the steps to reproduce?

好的。 1.sdmg2pb.py->torch2pb 函数中定义了5个输入，但是导出的onnx中只有4个。 2.参照mmocr的实现，代码里有些对tensor的slice操作，在转onnx的时候会引发 warning。

Bourne-M commented 2 years ago

@AllentDan could you help me

MohamedElghazi01 commented 6 months ago

Nov 12, 2021

Hi please did you put the script of converting the sdgmr model ?

MohamedElghazi01 commented 6 months ago

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Sorry for late reply tofuai. Give me few days, i will raise a PR. Thanks.

hi Hi please could you put the script of converting the sdgmr model ?