fine-tune sgpt-bloom-7b1-msmarco oom

Muennighoff / sgpt

SGPT: GPT Sentence Embeddings for Semantic Search

https://arxiv.org/abs/2202.08904

MIT License

841 stars 51 forks source link

fine-tune sgpt-bloom-7b1-msmarco oom #32

Open wing7171 opened 1 year ago

wing7171 commented 1 year ago

Hi, I have problem in fine-tunning sgpt-bloom-7b1-msmarco because of oom error, could you please share how you do contrasive fine-tuning on bloom-7b1? (I think distributed training is needed, but I failed ..)

Muennighoff commented 1 year ago

The command i used is here: https://huggingface.co/bigscience/sgpt-bloom-7b1-msmarco I ran on 8 A100 GPUs w/ 80GB I think