search

salesforce / awd-lstm-lm

LSTM and QRNN Language Model Toolkit for PyTorch
BSD 3-Clause "New" or "Revised" License
1.96k stars 488 forks source link

val_loss suddenly increases to a larger number(from ~5 to more than 10000) #121

Closed xuuuluuu closed 4 years ago

xuuuluuu commented 4 years ago

Hi, thanks a lot for this awesome Toolkit!

When I try to run the below command (I only change the original 3 layers to 1 layer LSTM),

python main.py --batch_size 20 --data data/penn --dropouti 0.4 --dropouth 0.25 --seed 141 --epoch 500 --save PTB.pt --nlayers 1

Such error triggered at certain(16th) epoch:

File "main.py", line 267, in epoch, (time.time() - epoch_start_time), val_loss, math.exp(val_loss), val_loss / math.log(2))) OverflowError: math range error

My initial found is that the val_loss suddenly increases to a larger number(from ~5 to more than 10000), which results in such error. However, I am not sure what cause this sudden huge increment.

xuuuluuu commented 4 years ago

looks like pytorch version issue