search

QingyaoAi / Deep-Listwise-Context-Model-for-Ranking-Refinement

A Tensorflow implementation of the Deep Listwise Context Model (DLCM) for ranking refinement.
Apache License 2.0
134 stars 56 forks source link

question about perplexity in training #4

Closed wintersurvival closed 4 years ago

wintersurvival commented 5 years ago

Hi QingyaoAi, Thanks for releasing code. I followed the steps in ReadMe. The training perplexity is decreasing and the eval perplexity is increasing in training process. Is there any problems in my experiment? Thanks.

global step 400 learning rate 0.5000 step-time 1.64 perplexity 7.67 eval: perplexity 7.56 global step 600 learning rate 0.5000 step-time 1.64 perplexity 7.62 eval: perplexity 7.57 global step 800 learning rate 0.5000 step-time 1.64 perplexity 7.57 eval: perplexity 7.50 global step 1000 learning rate 0.5000 step-time 1.63 perplexity 7.55 eval: perplexity 7.54 global step 1200 learning rate 0.5000 step-time 1.63 perplexity 7.49 eval: perplexity 7.53 global step 1400 learning rate 0.5000 step-time 1.63 perplexity 7.48 eval: perplexity 7.50 global step 1600 learning rate 0.5000 step-time 1.63 perplexity 7.48 eval: perplexity 7.51 global step 1800 learning rate 0.5000 step-time 1.64 perplexity 7.46 eval: perplexity 7.63 global step 2000 learning rate 0.5000 step-time 1.63 perplexity 7.38 eval: perplexity 7.50 global step 2200 learning rate 0.5000 step-time 1.63 perplexity 7.41 eval: perplexity 7.49 global step 2400 learning rate 0.5000 step-time 1.64 perplexity 7.41 eval: perplexity 7.55 global step 2600 learning rate 0.5000 step-time 1.64 perplexity 7.38 eval: perplexity 7.48 global step 2800 learning rate 0.5000 step-time 1.64 perplexity 7.33 eval: perplexity 7.51 global step 3000 learning rate 0.5000 step-time 1.64 perplexity 7.31 eval: perplexity 7.49 global step 3200 learning rate 0.5000 step-time 1.64 perplexity 7.27 eval: perplexity 7.61 global step 3400 learning rate 0.5000 step-time 1.64 perplexity 7.26 eval: perplexity 7.53 global step 3600 learning rate 0.5000 step-time 1.64 perplexity 7.23 eval: perplexity 7.51 global step 3800 learning rate 0.5000 step-time 1.63 perplexity 7.20 eval: perplexity 7.51 global step 4000 learning rate 0.5000 step-time 1.63 perplexity 7.23 eval: perplexity 7.49 global step 4200 learning rate 0.5000 step-time 1.63 perplexity 7.15 eval: perplexity 7.58 global step 4400 learning rate 0.5000 step-time 1.63 perplexity 7.10 eval: perplexity 7.50 global step 4600 learning rate 0.5000 step-time 1.64 perplexity 7.05 eval: perplexity 7.67 global step 4800 learning rate 0.5000 step-time 1.64 perplexity 6.98 eval: perplexity 7.72 global step 5000 learning rate 0.5000 step-time 1.63 perplexity 6.95 eval: perplexity 7.58 global step 5200 learning rate 0.5000 step-time 1.63 perplexity 6.90 eval: perplexity 7.60 global step 5400 learning rate 0.5000 step-time 1.63 perplexity 6.83 eval: perplexity 7.64 global step 5600 learning rate 0.5000 step-time 1.64 perplexity 6.77 eval: perplexity 7.61 global step 5800 learning rate 0.5000 step-time 1.64 perplexity 6.71 eval: perplexity 7.74 global step 6000 learning rate 0.5000 step-time 1.64 perplexity 6.64 eval: perplexity 7.77 global step 6200 learning rate 0.5000 step-time 1.64 perplexity 6.57 eval: perplexity 7.76 global step 6400 learning rate 0.5000 step-time 1.64 perplexity 6.53 eval: perplexity 7.80 global step 6600 learning rate 0.5000 step-time 1.64 perplexity 6.43 eval: perplexity 7.94 global step 6800 learning rate 0.5000 step-time 1.64 perplexity 6.40 eval: perplexity 7.96 global step 7000 learning rate 0.5000 step-time 1.64 perplexity 6.34 eval: perplexity 7.91 global step 7200 learning rate 0.5000 step-time 1.64 perplexity 6.30 eval: perplexity 7.98 global step 7400 learning rate 0.5000 step-time 1.64 perplexity 6.22 eval: perplexity 8.00 global step 7600 learning rate 0.5000 step-time 1.64 perplexity 6.17 eval: perplexity 8.06 global step 7800 learning rate 0.5000 step-time 1.64 perplexity 6.11 eval: perplexity 8.09 global step 8000 learning rate 0.5000 step-time 1.64 perplexity 6.05 eval: perplexity 8.08 global step 8200 learning rate 0.5000 step-time 1.64 perplexity 6.02 eval: perplexity 8.11 global step 8400 learning rate 0.5000 step-time 1.64 perplexity 5.98 eval: perplexity 8.11 global step 8600 learning rate 0.5000 step-time 1.64 perplexity 5.95 eval: perplexity 8.18 global step 8800 learning rate 0.5000 step-time 1.64 perplexity 5.91 eval: perplexity 8.23 global step 9000 learning rate 0.5000 step-time 1.64 perplexity 5.88 eval: perplexity 8.16 global step 9200 learning rate 0.5000 step-time 1.64 perplexity 5.85 eval: perplexity 8.28 global step 9400 learning rate 0.5000 step-time 1.64 perplexity 5.82 eval: perplexity 8.28 global step 9600 learning rate 0.5000 step-time 1.64 perplexity 5.80 eval: perplexity 8.25 global step 9800 learning rate 0.5000 step-time 1.64 perplexity 5.75 eval: perplexity 8.30 global step 10000 learning rate 0.5000 step-time 1.64 perplexity 5.70 eval: perplexity 8.29 global step 10200 learning rate 0.5000 step-time 1.64 perplexity 5.71 eval: perplexity 8.42 global step 10400 learning rate 0.5000 step-time 1.64 perplexity 5.70 eval: perplexity 8.47

QingyaoAi commented 5 years ago

It is weird to see that the eval perplexity keep increasing. In my case, it first decreased and then fluctuated around a certain number. In case you need, I attached the output of the model below:

Reading data in /mnt/scratch/aiqy/LSTM_rank/working/training_data/SVMrank/set1/RANK-CUT_40/ Rank list size 40 Creating 3 layers of 700 units. Loss Function is softmax Created model with fresh parameters. 0 global step 100 learning rate 0.5000 step-time 0.70 perplexity 8575.39 eval: perplexity 14.49 global step 200 learning rate 0.5000 step-time 0.69 perplexity 15.56 eval: perplexity 14.63 global step 300 learning rate 0.5000 step-time 0.69 perplexity 14.90 eval: perplexity 14.27 global step 400 learning rate 0.5000 step-time 0.69 perplexity 14.63 eval: perplexity 15.53 global step 500 learning rate 0.5000 step-time 0.69 perplexity 14.62 eval: perplexity 13.80 global step 600 learning rate 0.5000 step-time 0.68 perplexity 14.43 eval: perplexity 13.61 global step 700 learning rate 0.5000 step-time 0.68 perplexity 14.44 eval: perplexity 13.58 global step 800 learning rate 0.5000 step-time 0.68 perplexity 14.26 eval: perplexity 13.77 global step 900 learning rate 0.5000 step-time 0.68 perplexity 14.24 eval: perplexity 13.64 global step 1000 learning rate 0.5000 step-time 0.68 perplexity 14.21 eval: perplexity 13.76 global step 1100 learning rate 0.5000 step-time 0.68 perplexity 14.13 eval: perplexity 13.84 global step 1200 learning rate 0.5000 step-time 0.69 perplexity 14.24 eval: perplexity 13.52 global step 1300 learning rate 0.4500 step-time 0.69 perplexity 14.23 eval: perplexity 13.44 global step 1400 learning rate 0.4500 step-time 0.69 perplexity 14.13 eval: perplexity 13.51 global step 1500 learning rate 0.4500 step-time 0.69 perplexity 13.97 eval: perplexity 13.60 global step 1600 learning rate 0.4500 step-time 0.68 perplexity 14.04 eval: perplexity 13.52 global step 1700 learning rate 0.4500 step-time 0.68 perplexity 14.08 eval: perplexity 13.47 global step 1800 learning rate 0.4500 step-time 0.68 perplexity 14.10 eval: perplexity 13.41 global step 1900 learning rate 0.4050 step-time 0.69 perplexity 14.02 eval: perplexity 13.57 global step 2000 learning rate 0.4050 step-time 0.69 perplexity 13.97 eval: perplexity 13.46 global step 2100 learning rate 0.4050 step-time 0.68 perplexity 13.90 eval: perplexity 13.48 global step 2200 learning rate 0.4050 step-time 0.68 perplexity 14.04 eval: perplexity 13.33 global step 2300 learning rate 0.3645 step-time 0.69 perplexity 13.99 eval: perplexity 13.42 global step 2400 learning rate 0.3645 step-time 0.69 perplexity 14.09 eval: perplexity 13.39 global step 2500 learning rate 0.3280 step-time 0.67 perplexity 13.79 eval: perplexity 13.62 global step 2600 learning rate 0.3280 step-time 0.66 perplexity 13.98 eval: perplexity 13.34 global step 2700 learning rate 0.3280 step-time 0.66 perplexity 13.76 eval: perplexity 13.95 global step 2800 learning rate 0.3280 step-time 0.66 perplexity 13.88 eval: perplexity 13.58 global step 2900 learning rate 0.3280 step-time 0.67 perplexity 13.93 eval: perplexity 13.51 global step 3000 learning rate 0.3280 step-time 0.67 perplexity 13.79 eval: perplexity 13.29 global step 3100 learning rate 0.3280 step-time 0.67 perplexity 13.75 eval: perplexity 13.28 global step 3200 learning rate 0.3280 step-time 0.67 perplexity 13.82 eval: perplexity 13.39 global step 3300 learning rate 0.3280 step-time 0.67 perplexity 13.87 eval: perplexity 13.59 global step 3400 learning rate 0.2952 step-time 0.67 perplexity 13.65 eval: perplexity 13.26 global step 3500 learning rate 0.2952 step-time 0.67 perplexity 13.79 eval: perplexity 13.45 global step 3600 learning rate 0.2952 step-time 0.66 perplexity 13.63 eval: perplexity 13.32 global step 3700 learning rate 0.2952 step-time 0.66 perplexity 13.72 eval: perplexity 13.18 global step 3800 learning rate 0.2952 step-time 0.66 perplexity 13.82 eval: perplexity 13.39 global step 3900 learning rate 0.2657 step-time 0.66 perplexity 13.66 eval: perplexity 13.48 global step 4000 learning rate 0.2657 step-time 0.66 perplexity 13.66 eval: perplexity 13.20 global step 4100 learning rate 0.2657 step-time 0.66 perplexity 13.69 eval: perplexity 13.56 global step 4200 learning rate 0.2657 step-time 0.66 perplexity 13.86 eval: perplexity 13.32 global step 4300 learning rate 0.2391 step-time 0.65 perplexity 13.58 eval: perplexity 13.38 global step 4400 learning rate 0.2391 step-time 0.66 perplexity 13.66 eval: perplexity 13.52 global step 4500 learning rate 0.2391 step-time 0.65 perplexity 13.67 eval: perplexity 13.26 global step 4600 learning rate 0.2391 step-time 0.66 perplexity 13.66 eval: perplexity 13.50 global step 4700 learning rate 0.2391 step-time 0.66 perplexity 13.56 eval: perplexity 13.43 global step 4800 learning rate 0.2391 step-time 0.66 perplexity 13.70 eval: perplexity 13.30 global step 4900 learning rate 0.2152 step-time 0.66 perplexity 13.58 eval: perplexity 13.35 global step 5000 learning rate 0.2152 step-time 0.66 perplexity 13.67 eval: perplexity 13.25 global step 5100 learning rate 0.2152 step-time 0.65 perplexity 13.61 eval: perplexity 13.13 global step 5200 learning rate 0.2152 step-time 0.67 perplexity 13.58 eval: perplexity 13.31 global step 5300 learning rate 0.2152 step-time 0.67 perplexity 13.48 eval: perplexity 13.46 global step 5400 learning rate 0.2152 step-time 0.67 perplexity 13.54 eval: perplexity 13.18 global step 5500 learning rate 0.2152 step-time 0.66 perplexity 13.53 eval: perplexity 13.17 global step 5600 learning rate 0.2152 step-time 0.67 perplexity 13.64 eval: perplexity 13.24 global step 5700 learning rate 0.1937 step-time 0.66 perplexity 13.57 eval: perplexity 13.50 global step 5800 learning rate 0.1937 step-time 0.67 perplexity 13.48 eval: perplexity 13.53 global step 5900 learning rate 0.1937 step-time 0.67 perplexity 13.45 eval: perplexity 13.23 global step 6000 learning rate 0.1937 step-time 0.67 perplexity 13.56 eval: perplexity 13.34 global step 6100 learning rate 0.1937 step-time 0.67 perplexity 13.34 eval: perplexity 13.35 global step 6200 learning rate 0.1937 step-time 0.67 perplexity 13.57 eval: perplexity 13.25 global step 6300 learning rate 0.1743 step-time 0.66 perplexity 13.50 eval: perplexity 13.43 global step 6400 learning rate 0.1743 step-time 0.66 perplexity 13.40 eval: perplexity 13.28 global step 6500 learning rate 0.1743 step-time 0.67 perplexity 13.42 eval: perplexity 13.16 global step 6600 learning rate 0.1743 step-time 0.67 perplexity 13.40 eval: perplexity 13.68 global step 6700 learning rate 0.1743 step-time 0.67 perplexity 13.32 eval: perplexity 13.34 global step 6800 learning rate 0.1743 step-time 0.66 perplexity 13.30 eval: perplexity 13.60 global step 6900 learning rate 0.1743 step-time 0.67 perplexity 13.27 eval: perplexity 13.31 global step 7000 learning rate 0.1743 step-time 0.66 perplexity 13.30 eval: perplexity 13.21 global step 7100 learning rate 0.1743 step-time 0.66 perplexity 13.36 eval: perplexity 13.23 global step 7200 learning rate 0.1569 step-time 0.67 perplexity 13.37 eval: perplexity 13.27 global step 7300 learning rate 0.1412 step-time 0.67 perplexity 13.22 eval: perplexity 13.23 global step 7400 learning rate 0.1412 step-time 0.66 perplexity 13.32 eval: perplexity 13.34 global step 7500 learning rate 0.1412 step-time 0.66 perplexity 13.26 eval: perplexity 13.26 global step 7600 learning rate 0.1412 step-time 0.66 perplexity 13.24 eval: perplexity 13.88 global step 7700 learning rate 0.1412 step-time 0.67 perplexity 13.32 eval: perplexity 13.35 global step 7800 learning rate 0.1271 step-time 0.66 perplexity 13.31 eval: perplexity 13.39 global step 7900 learning rate 0.1271 step-time 0.67 perplexity 13.29 eval: perplexity 13.32 global step 8000 learning rate 0.1271 step-time 0.66 perplexity 13.29 eval: perplexity 13.43 global step 8100 learning rate 0.1271 step-time 0.66 perplexity 13.19 eval: perplexity 13.21 global step 8200 learning rate 0.1271 step-time 0.67 perplexity 13.19 eval: perplexity 13.38 global step 8300 learning rate 0.1271 step-time 0.67 perplexity 13.10 eval: perplexity 13.46 global step 8400 learning rate 0.1271 step-time 0.67 perplexity 12.97 eval: perplexity 13.51 global step 8500 learning rate 0.1271 step-time 0.66 perplexity 13.02 eval: perplexity 13.27 global step 8600 learning rate 0.1271 step-time 0.67 perplexity 13.10 eval: perplexity 13.14 global step 8700 learning rate 0.1271 step-time 0.67 perplexity 13.20 eval: perplexity 13.50 global step 8800 learning rate 0.1144 step-time 0.67 perplexity 13.18 eval: perplexity 13.31 global step 8900 learning rate 0.1144 step-time 0.67 perplexity 13.09 eval: perplexity 13.46 global step 9000 learning rate 0.1144 step-time 0.67 perplexity 13.02 eval: perplexity 13.41 global step 9100 learning rate 0.1144 step-time 0.67 perplexity 13.11 eval: perplexity 13.48 global step 9200 learning rate 0.1144 step-time 0.69 perplexity 13.06 eval: perplexity 13.27 global step 9300 learning rate 0.1144 step-time 0.67 perplexity 12.93 eval: perplexity 13.28 global step 9400 learning rate 0.1144 step-time 0.68 perplexity 12.96 eval: perplexity 13.31 global step 9500 learning rate 0.1144 step-time 0.68 perplexity 13.05 eval: perplexity 13.27 global step 9600 learning rate 0.1144 step-time 0.67 perplexity 13.04 eval: perplexity 13.33 global step 9700 learning rate 0.1144 step-time 0.66 perplexity 12.87 eval: perplexity 13.50 global step 9800 learning rate 0.1144 step-time 0.65 perplexity 13.01 eval: perplexity 13.44 global step 9900 learning rate 0.1144 step-time 0.66 perplexity 13.06 eval: perplexity 13.73 global step 10000 learning rate 0.1029 step-time 0.67 perplexity 12.91 eval: perplexity 13.52 global step 10100 learning rate 0.1029 step-time 0.67 perplexity 12.95 eval: perplexity 13.21