search

UKPLab / sentence-transformers

Multilingual Sentence & Image Embeddings with BERT
https://www.SBERT.net
Apache License 2.0
14.48k stars 2.4k forks source link

RuntimeError: Placeholder storage has not been allocated on MPS device! #2800

Open JakeRutherford opened 2 weeks ago

JakeRutherford commented 2 weeks ago

I am running this basic training example on an Apple M3 Pro. I am using Python 3.10, sentence-transformers 3.0.1, accelerate 0.31 and torch 2.3.1.

from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses
from datasets import Dataset

model = SentenceTransformer("all-MiniLM-L6-v2")
train_dataset = Dataset.from_dict({
    "anchor": ["It's nice weather outside today.", "He drove to work."],
    "positive": ["It's so sunny.", "He took the car to the office."],
    "negative": ["It's quite rainy, sadly.", "She walked to the store."],
})

loss = losses.TripletLoss(model=model)

trainer = SentenceTransformerTrainer(
    model=model,
    train_dataset=train_dataset,
    loss=loss,
)
trainer.train()

I am getting this error:

poetry run python foo.py
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
  0%|                                                                                                                                                                                       | 0/3 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/Users/jakerutherford/Documents/vlexlabs-experiments/experiments/headnotes_embeddings/foo.py", line 18, in <module>
    trainer.train()
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1932, in train
    return inner_training_loop(
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2268, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 3307, in training_step
    loss = self.compute_loss(model, inputs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/sentence_transformers/trainer.py", line 329, in compute_loss
    loss = loss_fn(features, labels)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/sentence_transformers/losses/TripletLoss.py", line 79, in forward
    reps = [self.model(sentence_feature)["sentence_embedding"] for sentence_feature in sentence_features]
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/sentence_transformers/losses/TripletLoss.py", line 79, in <listcomp>
    reps = [self.model(sentence_feature)["sentence_embedding"] for sentence_feature in sentence_features]
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
    input = module(input)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 118, in forward
    output_states = self.auto_model(**trans_features, return_dict=False)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1077, in forward
    embedding_output = self.embeddings(
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 210, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/Users/jakerutherford/Library/Caches/pypoetry/virtualenvs/headnotes-embeddings-_XuWiqEr-py3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 2233, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Placeholder storage has not been allocated on MPS device!

How can this be resolved?

tomaarsen commented 2 weeks ago

Hello!

I don't think I've seen this issue before, but a bit more searching makes me suggest that perhaps you have to use use_mps_device=True in your SentenceTransformerTrainingArguments:

from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, SentenceTransformerTrainingArguments, losses
from datasets import Dataset

model = SentenceTransformer("all-MiniLM-L6-v2")
train_dataset = Dataset.from_dict({
    "anchor": ["It's nice weather outside today.", "He drove to work."],
    "positive": ["It's so sunny.", "He took the car to the office."],
    "negative": ["It's quite rainy, sadly.", "She walked to the store."],
})

loss = losses.TripletLoss(model=model)

args = SentenceTransformerTrainingArguments(
    output_dir="test_trainer",
    use_mps_device=True,
)

trainer = SentenceTransformerTrainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    loss=loss,
)
trainer.train()
JakeRutherford commented 2 weeks ago

Hello @tomaarsen !

Unfortunately adding use_mps_device=True to the SentenceTransformerTrainingArguments still gives the same error 🤔:

RuntimeError: Placeholder storage has not been allocated on MPS device!

tomaarsen commented 2 weeks ago

Hmm, could you print the model.device after it has been loaded? If it's not "mps", could you move the model with model.to("mps")? I'm able to find some threads online when searching for RuntimeError: Placeholder storage has not been allocated on MPS device!, but I'm not quite sure which might help.

wanggz commented 1 week ago

Hi, I have the same error, when I print the model.device after model loaded is "mps:0".

here is the version of package:

accelerate 0.32.1 transformers 4.42.3 sentence-transformers 3.0.1 torch 2.3.1

and I use Python 3.11.5