Question about embedding dropout vs lockeddropout

salesforce / awd-lstm-lm

LSTM and QRNN Language Model Toolkit for PyTorch

BSD 3-Clause "New" or "Revised" License

1.96k stars 488 forks source link

Closed YontiLevin closed 6 years ago

YontiLevin commented 6 years ago

why do you apply embedding dropout in line 70 of the model.py and then apply LockedDropout in line 73?

doesn't both functions have the same functionality regrading the dropout?

is it equivalent to applying the embedding dropout with an higher rate?

many thanks

YontiLevin commented 6 years ago

never mind. figured it

embedding dropouts drop entire words the locked dropouts on it drop values inside word embeddings