search

salesforce / awd-lstm-lm

LSTM and QRNN Language Model Toolkit for PyTorch
BSD 3-Clause "New" or "Revised" License
1.96k stars 488 forks source link

UserWarning: RNN module weights are not part of single contiguous chunk of memory. and how to generate probability of a setence #7

Closed dixiematt8 closed 7 years ago

dixiematt8 commented 7 years ago

I am getting the following warning:

UserWarning: RNN module weights are not part of a single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().

I am using pytorch 0.20 and python 3.5

Also, how do you generate a probability of a given sentence?

Smerity commented 7 years ago

Hi @dixiematt8,

This is an expected UserWarning and isn't a problem. As the weights are changed before each call of the LSTM, they'd need to be compacted anyway.

Re: probability of a given sentence, that is not defined. As language is infinite in the sentences it can generate, there's no well defined probability distribution. You can see which sentences may be more of less likely but that's about the best I could suggest.

For that you'd multiply the probabilities (or sum the log probabilities for a numerically more stable approach) of each of the words according the model's prediction and select the sentence with the larger probability.

Hope that helps!