search

shibing624 / text2vec

text2vec, text to vector. 文本向量表征工具，把文本转化为向量矩阵，实现了Word2Vec、RankBM25、Sentence-BERT、CoSENT等文本表征、文本相似度计算模型，开箱即用。
https://pypi.org/project/text2vec/
Apache License 2.0
4.48k stars 396 forks source link

是否支持模型加速 #56

Closed flydsc closed 1 year ago

flydsc commented 1 year ago

请教目前是否可以支持模型加速部署的链路？或者可以用hugging face的API来做ONNX部署？

shibing624 commented 1 year ago

可以onnx。

flydsc commented 1 year ago

感谢回复！想请教一下目前保存成sbert的格式的话，有没有成熟的转onnx来做的方案呢？

shibing624 commented 1 year ago

有 https://github.com/yuanzhoulvpi2017/quick_sentence_transformers ,用的onnx， 你可以借鉴。

flydsc commented 1 year ago

找到一个即插即用的库，供后来大家参考

实测量化效果会有影响，供大家参考：

https://github.com/Pandora-Intelligence/fast-sentence-transformers

用例：


from fast_sentence_transformers import FastSentenceTransformer as SentenceTransformer

# use any sentence-transformer
encoder = SentenceTransformer("all-MiniLM-L6-v2", device="cpu", quantize=True)

encoder.encode("Hello hello, hey, hello hello")
encoder.encode(["Life is too short to eat bad food!"] * 2)