search

UKPLab / sentence-transformers

State-of-the-Art Text Embeddings
https://www.sbert.net
Apache License 2.0
15.21k stars 2.47k forks source link

Memory issue #1469

Open ErfolgreichCharismatisch opened 2 years ago

ErfolgreichCharismatisch commented 2 years ago

Loading from sentence_transformers import SentenceTransformer, util costs 228 additional MB in memory, model = SentenceTransformer('path\\to\\sentence_transformers\\sentence-transformers_msmarco-distilbert-multilingual-en-de-v2-tmp-trained-scratch', device='cuda') costs 1.1 GB.

with open(pkl, "rb") as fIn:
    stored_data = pickle.load(fIn)
    doc_emb = stored_data['doc_emb']

costs another 971 MB, query_emb = model.encode(query, batch_size=6) another 867 MB.

How to organize memory more efficiently?

nreimers commented 2 years ago

Transformers model costs quite a lot of memory. You can try to use quantization to reduce the model size