search

clear-nus / edc

MIT License
32 stars 4 forks source link

Can edc use other embedding tokenizer #3

Closed hanw closed 1 month ago

hanw commented 1 month ago

intfloat/e5-mistral-7b-instruct is hard-coded in edc_framework.py, can it be replaced by other embedding model?

bzhangj13zzz commented 1 month ago

Yes, but we only tested with this model which is the state-of-the-art embedding model, I would recommend checking out sentence transformers. https://sbert.net/ It may also solve the memory issues as these models are much much smaller.