search

MaartenGr / KeyBERT

Minimal keyword extraction with BERT
https://MaartenGr.github.io/KeyBERT/
MIT License
3.5k stars 345 forks source link

Support for custom ONNX optimized models #48

Closed sujithjoseph closed 2 years ago

sujithjoseph commented 3 years ago

How can i pass a custom model url (say one optimized for inference for onnx / tensorRT).

MaartenGr commented 3 years ago

This is currently not supported as Flair, which is the main method for using embedding models, uses hugginface transformers which is typically not meant for importing onnx structures in part due to their tokenization structure. Exporting to onnx is possible however.

sujithjoseph commented 3 years ago

This is available in sentencetransformers - https://github.com/UKPLab/sentence-transformers/blob/72da7293186700bff4fed8e81adcfe40a03be1d2/examples/onnx_inference/onnx_inference.ipynb . We can load an onnx model for inference. With a local path to onnx exported file, it could be added as one more backend . Did test it with tinyBERT exported onnx model, but didnt see much performance gain on a V100. yet to try on a T4 GPU.

MaartenGr commented 3 years ago

Thank you for the link! I did not realize it was already available in sentence-transformers. The main difficulty though will be loading in custom ONNX models that require some processing before it can be used in sentence-transformers.

sujithjoseph commented 3 years ago

I am thinking of adding it as a custom backend. i didnt see this before - https://maartengr.github.io/KeyBERT/guides/embeddings.html#custom-backend

MaartenGr commented 3 years ago

Yes, if you can create a custom backend that loads in using the sentence-transformer then that would solve your issue. It would also be nice if that backend could then be added to KeyBERT but that would also depend on dependencies and implementation.

MaartenGr commented 2 years ago

Since this issue has been a while without activity, I'll be closing it for now. However, if you are still experiencing the issue or want to discuss it further, let me know!

sujithjoseph commented 2 years ago

Sure, Will add a PR later to add this back to KeyBERT. -Sujith

On Mon, Oct 25, 2021 at 11:20 PM Maarten Grootendorst < @.***> wrote:

Since this issue has been a while without activity, I'll be closing it for now. However, if you are still experiencing the issue or want to discuss it further, let me know!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MaartenGr/KeyBERT/issues/48#issuecomment-951598922, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABC7JFUBPL5MN7KWBLXHNTDUIZJC5ANCNFSM5ABWWQNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Matthieu-Tinycoaching commented 2 years ago

Hi @MaartenGr @sujithjoseph did you manage to deal with ONNX backend?