facebookresearch / InferSent

InferSent sentence embeddings
Other
2.28k stars 471 forks source link

RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 24776638464 bytes. Error code 12 (Cannot allocate memory) #144

Closed AFAgarap closed 3 years ago

AFAgarap commented 3 years ago

I tried the steps in README.md to encode the text data from sklearn.datasets.fetch_20newsgroups, and then I encountered this error.

RuntimeError                              Traceback (most recent call last)
<ipython-input-31-037d6b4d5107> in <module>
----> 1 embeddings = infersent.encode(features, tokenize=True)

~/Projects/sentence-embeddings/InferSent/models.py in encode(self, sentences, bsize, tokenize, verbose)
    220                 batch = batch.cuda()
    221             with torch.no_grad():
--> 222                 batch = self.forward((batch, lengths[stidx:stidx + bsize])).data.cpu().numpy()
    223             embeddings.append(batch)
    224         embeddings = np.vstack(embeddings)

~/Projects/sentence-embeddings/InferSent/models.py in forward(self, sent_tuple)
     63         # Handling padding in Recurrent Networks
     64         sent_packed = nn.utils.rnn.pack_padded_sequence(sent, sent_len_sorted)
---> 65         sent_output = self.enc_lstm(sent_packed)[0]  # seqlen x batch x 2*nhid
     66         sent_output = nn.utils.rnn.pad_packed_sequence(sent_output)[0]
     67

/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.8/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    577                               self.dropout, self.training, self.bidirectional, self.batch_first)
    578         else:
--> 579             result = _VF.lstm(input, batch_sizes, hx, self._flat_weights, self.bias,
    580                               self.num_layers, self.dropout, self.training, self.bidirectional)
    581         output = result[0]

RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 24776638464 bytes. Error code 12 (Cannot allocate memory)

I hope someone can help me resolve this issue. Thank you.