search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.54k stars 182 forks source link

Support multi-gpu training? #90

Closed jordane95 closed 2 years ago

jordane95 commented 2 years ago

Hello,

I'm trying to use your code to train a bi-encoder on msmarco, but it seems that using a comparable batch size exceeds my GPU memory. Does your code support multi-gpu training?

ahadda5 commented 2 years ago

@jordane95 how are you moving the data to GPU. Did you move data to GPU? train_dataset.to(device)

I don't see torch in that code , and that is the only way i know to use GPU

ahadda5 commented 2 years ago

My bad, cuda was not enabled and its seems BeIR uses it automatically if available

[2022-06-06 10:29:03] INFO [beir.generation.models.auto_model.init:16] Use pytorch device: cuda

Now back to your multi-Gpu question :)

thakur-nandan commented 2 years ago

Hi @jordane95 and @ahadda5, sadly the code present in the repository uses Sentence-Transformers as a framework where multi-GPU training is yet not available.

You can have a look at Tevatron in case you are interested in multi-GPU training.

Kind Regards, Nandan Thakur

jordane95 commented 2 years ago

Thanks! I will check it out.

rahmanidashti commented 1 year ago

jordane95

Hi @jordane95, could you solve this issue? I am also running dense models on MSMARCO but it exceeds memory during encoding corpus.

jordane95 commented 1 year ago

jordane95

Hi @jordane95, could you solve this issue? I am also running dense models on MSMARCO but it exceeds memory during encoding corpus.

Hi @rahmanidashti , may be you could try the tevatron toolkit which supports chunked corpus encoding