search

microsoft / LightLDA

Scalable, fast, and lightweight system for large-scale topic modeling
http://www.dmtk.io
MIT License
842 stars 234 forks source link

Likelihood not converging #13

Closed ross85 closed 8 years ago

ross85 commented 8 years ago

Hi, I am trying to run one of the examples (nytimes.sh). I have modified only the number of local workers (set to 6) and I launched the script out of the box.

It works, but the likelihood does not seem to converge like in the WWW'15 paper. Doc likelihood seems to be fine, word likelihood is positive and it increases, while Normalized likelihood does not move so much. Here an extract from the log:

[INFO] [2015-11-26 18:55:17] doc likelihood : -6.422044e+08
[INFO] [2015-11-26 18:55:18] word likelihood : 5.328907e+08
[INFO] [2015-11-26 18:55:18] Normalized likelihood : -1.561648e+09
[INFO] [2015-11-26 18:58:36] doc likelihood : -6.044065e+08
[INFO] [2015-11-26 18:58:37] word likelihood : 5.660771e+08
[INFO] [2015-11-26 18:58:37] Normalized likelihood : -1.562606e+09
[INFO] [2015-11-26 19:01:52] doc likelihood : -5.938570e+08
[INFO] [2015-11-26 19:01:53] word likelihood : 5.854258e+08
[INFO] [2015-11-26 19:01:53] Normalized likelihood : -1.564632e+09
[INFO] [2015-11-26 19:05:00] doc likelihood : -5.767895e+08
[INFO] [2015-11-26 19:05:01] word likelihood : 6.065751e+08
[INFO] [2015-11-26 19:05:01] Normalized likelihood : -1.567653e+09
[INFO] [2015-11-26 19:08:03] doc likelihood : -5.527261e+08
[INFO] [2015-11-26 19:08:04] word likelihood : 6.345852e+08
[INFO] [2015-11-26 19:08:04] Normalized likelihood : -1.571537e+09
[INFO] [2015-11-26 19:10:59] doc likelihood : -5.192143e+08
[INFO] [2015-11-26 19:10:59] word likelihood : 6.688301e+08
[INFO] [2015-11-26 19:10:59] Normalized likelihood : -1.575203e+09
[INFO] [2015-11-26 19:13:52] doc likelihood : -4.839103e+08
[INFO] [2015-11-26 19:13:53] word likelihood : 7.142310e+08
[INFO] [2015-11-26 19:13:53] Normalized likelihood : -1.579249e+09
[INFO] [2015-11-26 19:16:27] doc likelihood : -4.594354e+08
[INFO] [2015-11-26 19:16:28] word likelihood : 7.576918e+08
[INFO] [2015-11-26 19:16:28] Normalized likelihood : -1.582640e+09
[INFO] [2015-11-26 19:18:54] doc likelihood : -4.451361e+08
[INFO] [2015-11-26 19:18:54] word likelihood : 7.904300e+08
[INFO] [2015-11-26 19:18:54] Normalized likelihood : -1.584283e+09
[INFO] [2015-11-26 19:21:17] doc likelihood : -4.367304e+08
[INFO] [2015-11-26 19:21:18] word likelihood : 8.138167e+08
[INFO] [2015-11-26 19:21:18] Normalized likelihood : -1.584595e+09
[INFO] [2015-11-26 19:23:40] doc likelihood : -4.314949e+08
[INFO] [2015-11-26 19:23:40] word likelihood : 8.308841e+08
[INFO] [2015-11-26 19:23:40] Normalized likelihood : -1.584257e+09
[INFO] [2015-11-26 19:26:01] doc likelihood : -4.280392e+08
[INFO] [2015-11-26 19:26:01] word likelihood : 8.438210e+08
[INFO] [2015-11-26 19:26:01] Normalized likelihood : -1.583670e+09
[INFO] [2015-11-26 19:28:15] doc likelihood : -4.257501e+08
[INFO] [2015-11-26 19:28:15] word likelihood : 8.539954e+08
[INFO] [2015-11-26 19:28:15] Normalized likelihood : -1.583021e+09
[INFO] [2015-11-26 19:30:33] doc likelihood : -4.240845e+08
[INFO] [2015-11-26 19:30:33] word likelihood : 8.621267e+08
[INFO] [2015-11-26 19:30:33] Normalized likelihood : -1.582389e+09
[INFO] [2015-11-26 19:32:51] doc likelihood : -4.229418e+08
[INFO] [2015-11-26 19:32:52] word likelihood : 8.688454e+08
[INFO] [2015-11-26 19:32:52] Normalized likelihood : -1.581791e+09
[INFO] [2015-11-26 19:35:02] doc likelihood : -4.220603e+08
[INFO] [2015-11-26 19:35:02] word likelihood : 8.744806e+08
[INFO] [2015-11-26 19:35:02] Normalized likelihood : -1.581272e+09
[INFO] [2015-11-26 19:37:16] doc likelihood : -4.215044e+08
[INFO] [2015-11-26 19:37:17] word likelihood : 8.793446e+08
[INFO] [2015-11-26 19:37:17] Normalized likelihood : -1.580807e+09
[INFO] [2015-11-26 19:39:31] doc likelihood : -4.210534e+08
[INFO] [2015-11-26 19:39:32] word likelihood : 8.835320e+08
[INFO] [2015-11-26 19:39:32] Normalized likelihood : -1.580384e+09
[INFO] [2015-11-26 19:41:46] doc likelihood : -4.206959e+08
[INFO] [2015-11-26 19:41:46] word likelihood : 8.872202e+08
[INFO] [2015-11-26 19:41:46] Normalized likelihood : -1.580009e+09
[INFO] [2015-11-26 19:43:53] doc likelihood : -4.205036e+08
[INFO] [2015-11-26 19:43:54] word likelihood : 8.904672e+08
[INFO] [2015-11-26 19:43:54] Normalized likelihood : -1.579677e+09

Here the full log:

[INFO] [2015-11-26 18:54:21] INFO: block = 0, the number of slice = 1
[INFO] [2015-11-26 18:54:21] Server 0 starts: num_workers=1 endpoint=inproc://server
[INFO] [2015-11-26 18:54:21] Server 0: Worker registratrion completed: workers=1 trainers=6 servers=1
[INFO] [2015-11-26 18:54:21] Rank 0/1: Multiverso initialized successfully.
[INFO] [2015-11-26 18:54:21] Rank 0/1: Begin of configuration and initialization.
[INFO] [2015-11-26 18:54:36] Rank 0/1: End of configration and initialization.
[INFO] [2015-11-26 18:54:36] Rank 0/1: Begin of training.
[DEBUG] [2015-11-26 18:54:36] Request params. start = 0, end = 101635
[INFO] [2015-11-26 18:54:38] Rank = 0, Iter = 0, Block = 0, Slice = 0
[INFO] [2015-11-26 18:54:38] Rank = 0, Alias Time used: 2.29 s 
[INFO] [2015-11-26 18:55:10] Rank = 0, Training Time used: 198.78 s 
[INFO] [2015-11-26 18:55:10] Rank = 0, sampling throughput: 83392.330586 (tokens/thread/sec) 
[INFO] [2015-11-26 18:55:17] doc likelihood : -6.422044e+08
[INFO] [2015-11-26 18:55:18] word likelihood : 5.328907e+08
[INFO] [2015-11-26 18:55:18] Normalized likelihood : -1.561648e+09
[INFO] [2015-11-26 18:55:18] Rank = 0, Evaluation Time used: 37.24 s 
[DEBUG] [2015-11-26 18:55:18] Request params. start = 0, end = 101635
[INFO] [2015-11-26 18:55:21] Rank = 0, Iter = 1, Block = 0, Slice = 0
[INFO] [2015-11-26 18:55:22] Rank = 0, Alias Time used: 2.57 s 
[INFO] [2015-11-26 18:55:50] Rank = 0, Training Time used: 172.52 s 
[INFO] [2015-11-26 18:55:50] Rank = 0, sampling throughput: 96088.618782 (tokens/thread/sec) 
[DEBUG] [2015-11-26 18:55:52] Request params. start = 0, end = 101635
[INFO] [2015-11-26 18:56:02] Rank = 0, Iter = 2, Block = 0, Slice = 0
[INFO] [2015-11-26 18:56:03] Rank = 0, Alias Time used: 3.10 s 
[INFO] [2015-11-26 18:56:32] Rank = 0, Training Time used: 177.80 s 
[INFO] [2015-11-26 18:56:32] Rank = 0, sampling throughput: 93233.830746 (tokens/thread/sec) 
[DEBUG] [2015-11-26 18:56:32] Request params. start = 0, end = 101635
[INFO] [2015-11-26 18:56:40] Rank = 0, Iter = 3, Block = 0, Slice = 0
[INFO] [2015-11-26 18:56:40] Rank = 0, Alias Time used: 2.44 s 
[INFO] [2015-11-26 18:57:10] Rank = 0, Training Time used: 174.81 s 
[INFO] [2015-11-26 18:57:10] Rank = 0, sampling throughput: 94764.054198 (tokens/thread/sec) 
[DEBUG] [2015-11-26 18:57:11] Request params. start = 0, end = 101635
[INFO] [2015-11-26 18:57:19] Rank = 0, Iter = 4, Block = 0, Slice = 0
[INFO] [2015-11-26 18:57:20] Rank = 0, Alias Time used: 2.61 s 
[INFO] [2015-11-26 18:57:52] Rank = 0, Training Time used: 182.20 s 
[INFO] [2015-11-26 18:57:52] Rank = 0, sampling throughput: 90941.209369 (tokens/thread/sec) 
[DEBUG] [2015-11-26 18:57:52] Request params. start = 0, end = 101635
[INFO] [2015-11-26 18:58:00] Rank = 0, Iter = 5, Block = 0, Slice = 0
[INFO] [2015-11-26 18:58:00] Rank = 0, Alias Time used: 2.44 s 
[INFO] [2015-11-26 18:58:29] Rank = 0, Training Time used: 176.67 s 
[INFO] [2015-11-26 18:58:29] Rank = 0, sampling throughput: 93767.307385 (tokens/thread/sec) 
[INFO] [2015-11-26 18:58:36] doc likelihood : -6.044065e+08
[INFO] [2015-11-26 18:58:37] word likelihood : 5.660771e+08
[INFO] [2015-11-26 18:58:37] Normalized likelihood : -1.562606e+09
[INFO] [2015-11-26 18:58:37] Rank = 0, Evaluation Time used: 40.60 s 
[DEBUG] [2015-11-26 18:58:37] Request params. start = 0, end = 101635
[INFO] [2015-11-26 18:58:41] Rank = 0, Iter = 6, Block = 0, Slice = 0
[INFO] [2015-11-26 18:58:41] Rank = 0, Alias Time used: 2.54 s 
[INFO] [2015-11-26 18:59:11] Rank = 0, Training Time used: 175.08 s 
[INFO] [2015-11-26 18:59:11] Rank = 0, sampling throughput: 94682.829457 (tokens/thread/sec) 
[DEBUG] [2015-11-26 18:59:11] Request params. start = 0, end = 101635
[INFO] [2015-11-26 18:59:21] Rank = 0, Iter = 7, Block = 0, Slice = 0
[INFO] [2015-11-26 18:59:22] Rank = 0, Alias Time used: 2.50 s 
[INFO] [2015-11-26 18:59:51] Rank = 0, Training Time used: 174.15 s 
[INFO] [2015-11-26 18:59:51] Rank = 0, sampling throughput: 95190.270547 (tokens/thread/sec) 
[DEBUG] [2015-11-26 18:59:52] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:00:00] Rank = 0, Iter = 8, Block = 0, Slice = 0
[INFO] [2015-11-26 19:00:00] Rank = 0, Alias Time used: 2.70 s 
[INFO] [2015-11-26 19:00:30] Rank = 0, Training Time used: 179.12 s 
[INFO] [2015-11-26 19:00:30] Rank = 0, sampling throughput: 92546.646382 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:00:30] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:00:39] Rank = 0, Iter = 9, Block = 0, Slice = 0
[INFO] [2015-11-26 19:00:39] Rank = 0, Alias Time used: 2.28 s 
[INFO] [2015-11-26 19:01:06] Rank = 0, Training Time used: 164.11 s 
[INFO] [2015-11-26 19:01:06] Rank = 0, sampling throughput: 101008.266256 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:01:10] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:01:15] Rank = 0, Iter = 10, Block = 0, Slice = 0
[INFO] [2015-11-26 19:01:16] Rank = 0, Alias Time used: 2.15 s 
[INFO] [2015-11-26 19:01:40] Rank = 0, Training Time used: 143.45 s 
[INFO] [2015-11-26 19:01:40] Rank = 0, sampling throughput: 115558.460082 (tokens/thread/sec) 
[INFO] [2015-11-26 19:01:52] doc likelihood : -5.938570e+08
[INFO] [2015-11-26 19:01:53] word likelihood : 5.854258e+08
[INFO] [2015-11-26 19:01:53] Normalized likelihood : -1.564632e+09
[INFO] [2015-11-26 19:01:53] Rank = 0, Evaluation Time used: 74.25 s 
[DEBUG] [2015-11-26 19:01:53] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:01:55] Rank = 0, Iter = 11, Block = 0, Slice = 0
[INFO] [2015-11-26 19:01:55] Rank = 0, Alias Time used: 2.05 s 
[INFO] [2015-11-26 19:02:25] Rank = 0, Training Time used: 176.92 s 
[INFO] [2015-11-26 19:02:25] Rank = 0, sampling throughput: 93695.741097 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:02:26] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:02:35] Rank = 0, Iter = 12, Block = 0, Slice = 0
[INFO] [2015-11-26 19:02:36] Rank = 0, Alias Time used: 2.25 s 
[INFO] [2015-11-26 19:03:05] Rank = 0, Training Time used: 174.86 s 
[INFO] [2015-11-26 19:03:05] Rank = 0, sampling throughput: 94802.995521 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:03:05] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:03:11] Rank = 0, Iter = 13, Block = 0, Slice = 0
[INFO] [2015-11-26 19:03:12] Rank = 0, Alias Time used: 2.10 s 
[INFO] [2015-11-26 19:03:40] Rank = 0, Training Time used: 170.69 s 
[INFO] [2015-11-26 19:03:40] Rank = 0, sampling throughput: 97116.254759 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:03:41] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:03:48] Rank = 0, Iter = 14, Block = 0, Slice = 0
[INFO] [2015-11-26 19:03:49] Rank = 0, Alias Time used: 2.44 s 
[INFO] [2015-11-26 19:04:16] Rank = 0, Training Time used: 165.70 s 
[INFO] [2015-11-26 19:04:16] Rank = 0, sampling throughput: 99987.432960 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:04:19] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:04:26] Rank = 0, Iter = 15, Block = 0, Slice = 0
[INFO] [2015-11-26 19:04:27] Rank = 0, Alias Time used: 2.42 s 
[INFO] [2015-11-26 19:04:55] Rank = 0, Training Time used: 169.51 s 
[INFO] [2015-11-26 19:04:55] Rank = 0, sampling throughput: 97792.991463 (tokens/thread/sec) 
[INFO] [2015-11-26 19:05:00] doc likelihood : -5.767895e+08
[INFO] [2015-11-26 19:05:01] word likelihood : 6.065751e+08
[INFO] [2015-11-26 19:05:01] Normalized likelihood : -1.567653e+09
[INFO] [2015-11-26 19:05:01] Rank = 0, Evaluation Time used: 34.50 s 
[DEBUG] [2015-11-26 19:05:01] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:05:07] Rank = 0, Iter = 16, Block = 0, Slice = 0
[INFO] [2015-11-26 19:05:08] Rank = 0, Alias Time used: 2.14 s 
[INFO] [2015-11-26 19:05:36] Rank = 0, Training Time used: 168.02 s 
[INFO] [2015-11-26 19:05:36] Rank = 0, sampling throughput: 98663.274813 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:05:38] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:05:45] Rank = 0, Iter = 17, Block = 0, Slice = 0
[INFO] [2015-11-26 19:05:46] Rank = 0, Alias Time used: 2.00 s 
[INFO] [2015-11-26 19:06:14] Rank = 0, Training Time used: 172.06 s 
[INFO] [2015-11-26 19:06:14] Rank = 0, sampling throughput: 96346.686169 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:06:14] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:06:22] Rank = 0, Iter = 18, Block = 0, Slice = 0
[INFO] [2015-11-26 19:06:22] Rank = 0, Alias Time used: 1.74 s 
[INFO] [2015-11-26 19:06:53] Rank = 0, Training Time used: 183.90 s 
[INFO] [2015-11-26 19:06:53] Rank = 0, sampling throughput: 90142.883567 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:06:53] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:06:55] Rank = 0, Iter = 19, Block = 0, Slice = 0
[INFO] [2015-11-26 19:06:56] Rank = 0, Alias Time used: 2.25 s 
[INFO] [2015-11-26 19:07:26] Rank = 0, Training Time used: 178.41 s 
[INFO] [2015-11-26 19:07:26] Rank = 0, sampling throughput: 92913.800145 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:07:26] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:07:29] Rank = 0, Iter = 20, Block = 0, Slice = 0
[INFO] [2015-11-26 19:07:29] Rank = 0, Alias Time used: 1.88 s 
[INFO] [2015-11-26 19:07:52] Rank = 0, Training Time used: 139.02 s 
[INFO] [2015-11-26 19:07:52] Rank = 0, sampling throughput: 119193.773232 (tokens/thread/sec) 
[INFO] [2015-11-26 19:08:03] doc likelihood : -5.527261e+08
[INFO] [2015-11-26 19:08:04] word likelihood : 6.345852e+08
[INFO] [2015-11-26 19:08:04] Normalized likelihood : -1.571537e+09
[INFO] [2015-11-26 19:08:04] Rank = 0, Evaluation Time used: 63.24 s 
[DEBUG] [2015-11-26 19:08:04] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:08:08] Rank = 0, Iter = 21, Block = 0, Slice = 0
[INFO] [2015-11-26 19:08:09] Rank = 0, Alias Time used: 1.93 s 
[INFO] [2015-11-26 19:08:36] Rank = 0, Training Time used: 165.63 s 
[INFO] [2015-11-26 19:08:36] Rank = 0, sampling throughput: 100086.078133 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:08:39] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:08:42] Rank = 0, Iter = 22, Block = 0, Slice = 0
[INFO] [2015-11-26 19:08:43] Rank = 0, Alias Time used: 2.00 s 
[INFO] [2015-11-26 19:09:11] Rank = 0, Training Time used: 168.89 s 
[INFO] [2015-11-26 19:09:11] Rank = 0, sampling throughput: 98154.747588 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:09:13] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:09:19] Rank = 0, Iter = 23, Block = 0, Slice = 0
[INFO] [2015-11-26 19:09:19] Rank = 0, Alias Time used: 1.59 s 
[INFO] [2015-11-26 19:09:45] Rank = 0, Training Time used: 157.25 s 
[INFO] [2015-11-26 19:09:45] Rank = 0, sampling throughput: 105420.495884 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:09:48] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:09:50] Rank = 0, Iter = 24, Block = 0, Slice = 0
[INFO] [2015-11-26 19:09:51] Rank = 0, Alias Time used: 1.74 s 
[INFO] [2015-11-26 19:10:18] Rank = 0, Training Time used: 161.92 s 
[INFO] [2015-11-26 19:10:18] Rank = 0, sampling throughput: 102374.893736 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:10:19] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:10:25] Rank = 0, Iter = 25, Block = 0, Slice = 0
[INFO] [2015-11-26 19:10:26] Rank = 0, Alias Time used: 1.84 s 
[INFO] [2015-11-26 19:10:52] Rank = 0, Training Time used: 157.57 s 
[INFO] [2015-11-26 19:10:52] Rank = 0, sampling throughput: 105202.086087 (tokens/thread/sec) 
[INFO] [2015-11-26 19:10:59] doc likelihood : -5.192143e+08
[INFO] [2015-11-26 19:10:59] word likelihood : 6.688301e+08
[INFO] [2015-11-26 19:10:59] Normalized likelihood : -1.575203e+09
[INFO] [2015-11-26 19:10:59] Rank = 0, Evaluation Time used: 41.75 s 
[DEBUG] [2015-11-26 19:11:00] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:11:01] Rank = 0, Iter = 26, Block = 0, Slice = 0
[INFO] [2015-11-26 19:11:01] Rank = 0, Alias Time used: 1.97 s 
[INFO] [2015-11-26 19:11:24] Rank = 0, Training Time used: 138.78 s 
[INFO] [2015-11-26 19:11:24] Rank = 0, sampling throughput: 119448.079115 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:11:29] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:11:36] Rank = 0, Iter = 27, Block = 0, Slice = 0
[INFO] [2015-11-26 19:11:37] Rank = 0, Alias Time used: 2.40 s 
[INFO] [2015-11-26 19:12:13] Rank = 0, Training Time used: 184.30 s 
[INFO] [2015-11-26 19:12:13] Rank = 0, sampling throughput: 89946.089102 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:12:15] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:12:17] Rank = 0, Iter = 28, Block = 0, Slice = 0
[INFO] [2015-11-26 19:12:17] Rank = 0, Alias Time used: 2.10 s 
[INFO] [2015-11-26 19:12:45] Rank = 0, Training Time used: 164.67 s 
[INFO] [2015-11-26 19:12:45] Rank = 0, sampling throughput: 100667.571074 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:12:45] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:12:48] Rank = 0, Iter = 29, Block = 0, Slice = 0
[INFO] [2015-11-26 19:12:49] Rank = 0, Alias Time used: 1.67 s 
[INFO] [2015-11-26 19:13:17] Rank = 0, Training Time used: 165.13 s 
[INFO] [2015-11-26 19:13:17] Rank = 0, sampling throughput: 100342.776178 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:13:19] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:13:19] Rank = 0, Iter = 30, Block = 0, Slice = 0
[INFO] [2015-11-26 19:13:20] Rank = 0, Alias Time used: 1.66 s 
[INFO] [2015-11-26 19:13:45] Rank = 0, Training Time used: 155.96 s 
[INFO] [2015-11-26 19:13:45] Rank = 0, sampling throughput: 106286.736391 (tokens/thread/sec) 
[INFO] [2015-11-26 19:13:52] doc likelihood : -4.839103e+08
[INFO] [2015-11-26 19:13:53] word likelihood : 7.142310e+08
[INFO] [2015-11-26 19:13:53] Normalized likelihood : -1.579249e+09
[INFO] [2015-11-26 19:13:53] Rank = 0, Evaluation Time used: 36.55 s 
[DEBUG] [2015-11-26 19:13:53] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:13:54] Rank = 0, Iter = 31, Block = 0, Slice = 0
[INFO] [2015-11-26 19:13:54] Rank = 0, Alias Time used: 1.67 s 
[INFO] [2015-11-26 19:14:20] Rank = 0, Training Time used: 154.22 s 
[INFO] [2015-11-26 19:14:20] Rank = 0, sampling throughput: 107488.826353 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:14:20] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:14:24] Rank = 0, Iter = 32, Block = 0, Slice = 0
[INFO] [2015-11-26 19:14:24] Rank = 0, Alias Time used: 1.60 s 
[INFO] [2015-11-26 19:14:52] Rank = 0, Training Time used: 156.94 s 
[INFO] [2015-11-26 19:14:52] Rank = 0, sampling throughput: 105623.138350 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:14:57] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:14:58] Rank = 0, Iter = 33, Block = 0, Slice = 0
[INFO] [2015-11-26 19:14:58] Rank = 0, Alias Time used: 1.72 s 
[INFO] [2015-11-26 19:15:22] Rank = 0, Training Time used: 142.36 s 
[INFO] [2015-11-26 19:15:22] Rank = 0, sampling throughput: 116446.783711 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:15:26] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:15:27] Rank = 0, Iter = 34, Block = 0, Slice = 0
[INFO] [2015-11-26 19:15:28] Rank = 0, Alias Time used: 1.49 s 
[INFO] [2015-11-26 19:15:52] Rank = 0, Training Time used: 146.35 s 
[INFO] [2015-11-26 19:15:52] Rank = 0, sampling throughput: 113235.252901 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:15:54] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:15:56] Rank = 0, Iter = 35, Block = 0, Slice = 0
[INFO] [2015-11-26 19:15:56] Rank = 0, Alias Time used: 1.76 s 
[INFO] [2015-11-26 19:16:21] Rank = 0, Training Time used: 150.38 s 
[INFO] [2015-11-26 19:16:21] Rank = 0, sampling throughput: 110232.356935 (tokens/thread/sec) 
[INFO] [2015-11-26 19:16:27] doc likelihood : -4.594354e+08
[INFO] [2015-11-26 19:16:28] word likelihood : 7.576918e+08
[INFO] [2015-11-26 19:16:28] Normalized likelihood : -1.582640e+09
[INFO] [2015-11-26 19:16:28] Rank = 0, Evaluation Time used: 34.83 s 
[DEBUG] [2015-11-26 19:16:28] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:16:28] Rank = 0, Iter = 36, Block = 0, Slice = 0
[INFO] [2015-11-26 19:16:29] Rank = 0, Alias Time used: 1.78 s 
[INFO] [2015-11-26 19:16:52] Rank = 0, Training Time used: 139.90 s 
[INFO] [2015-11-26 19:16:52] Rank = 0, sampling throughput: 118453.417531 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:16:55] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:16:58] Rank = 0, Iter = 37, Block = 0, Slice = 0
[INFO] [2015-11-26 19:16:58] Rank = 0, Alias Time used: 1.47 s 
[INFO] [2015-11-26 19:17:21] Rank = 0, Training Time used: 142.79 s 
[INFO] [2015-11-26 19:17:21] Rank = 0, sampling throughput: 116058.543214 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:17:24] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:17:25] Rank = 0, Iter = 38, Block = 0, Slice = 0
[INFO] [2015-11-26 19:17:25] Rank = 0, Alias Time used: 1.69 s 
[INFO] [2015-11-26 19:17:46] Rank = 0, Training Time used: 122.43 s 
[INFO] [2015-11-26 19:17:46] Rank = 0, sampling throughput: 135402.404929 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:17:52] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:17:55] Rank = 0, Iter = 39, Block = 0, Slice = 0
[INFO] [2015-11-26 19:17:55] Rank = 0, Alias Time used: 1.92 s 
[INFO] [2015-11-26 19:18:16] Rank = 0, Training Time used: 125.18 s 
[INFO] [2015-11-26 19:18:16] Rank = 0, sampling throughput: 132423.392824 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:18:21] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:18:23] Rank = 0, Iter = 40, Block = 0, Slice = 0
[INFO] [2015-11-26 19:18:23] Rank = 0, Alias Time used: 1.90 s 
[INFO] [2015-11-26 19:18:49] Rank = 0, Training Time used: 153.46 s 
[INFO] [2015-11-26 19:18:49] Rank = 0, sampling throughput: 108023.534886 (tokens/thread/sec) 
[INFO] [2015-11-26 19:18:54] doc likelihood : -4.451361e+08
[INFO] [2015-11-26 19:18:54] word likelihood : 7.904300e+08
[INFO] [2015-11-26 19:18:54] Normalized likelihood : -1.584283e+09
[INFO] [2015-11-26 19:18:54] Rank = 0, Evaluation Time used: 27.21 s 
[DEBUG] [2015-11-26 19:18:54] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:18:55] Rank = 0, Iter = 41, Block = 0, Slice = 0
[INFO] [2015-11-26 19:18:55] Rank = 0, Alias Time used: 1.83 s 
[INFO] [2015-11-26 19:19:22] Rank = 0, Training Time used: 155.21 s 
[INFO] [2015-11-26 19:19:22] Rank = 0, sampling throughput: 106805.034757 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:19:23] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:19:24] Rank = 0, Iter = 42, Block = 0, Slice = 0
[INFO] [2015-11-26 19:19:24] Rank = 0, Alias Time used: 1.72 s 
[INFO] [2015-11-26 19:19:51] Rank = 0, Training Time used: 158.81 s 
[INFO] [2015-11-26 19:19:51] Rank = 0, sampling throughput: 104381.090406 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:19:52] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:19:53] Rank = 0, Iter = 43, Block = 0, Slice = 0
[INFO] [2015-11-26 19:19:53] Rank = 0, Alias Time used: 1.88 s 
[INFO] [2015-11-26 19:20:17] Rank = 0, Training Time used: 133.73 s 
[INFO] [2015-11-26 19:20:17] Rank = 0, sampling throughput: 123955.008691 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:20:19] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:20:21] Rank = 0, Iter = 44, Block = 0, Slice = 0
[INFO] [2015-11-26 19:20:21] Rank = 0, Alias Time used: 1.76 s 
[INFO] [2015-11-26 19:20:45] Rank = 0, Training Time used: 142.14 s 
[INFO] [2015-11-26 19:20:45] Rank = 0, sampling throughput: 116556.597077 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:20:46] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:20:47] Rank = 0, Iter = 45, Block = 0, Slice = 0
[INFO] [2015-11-26 19:20:47] Rank = 0, Alias Time used: 1.95 s 
[INFO] [2015-11-26 19:21:11] Rank = 0, Training Time used: 138.88 s 
[INFO] [2015-11-26 19:21:11] Rank = 0, sampling throughput: 119358.778462 (tokens/thread/sec) 
[INFO] [2015-11-26 19:21:17] doc likelihood : -4.367304e+08
[INFO] [2015-11-26 19:21:18] word likelihood : 8.138167e+08
[INFO] [2015-11-26 19:21:18] Normalized likelihood : -1.584595e+09
[INFO] [2015-11-26 19:21:18] Rank = 0, Evaluation Time used: 34.18 s 
[DEBUG] [2015-11-26 19:21:18] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:21:18] Rank = 0, Iter = 46, Block = 0, Slice = 0
[INFO] [2015-11-26 19:21:19] Rank = 0, Alias Time used: 2.07 s 
[INFO] [2015-11-26 19:21:44] Rank = 0, Training Time used: 139.06 s 
[INFO] [2015-11-26 19:21:44] Rank = 0, sampling throughput: 119130.069216 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:21:48] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:21:49] Rank = 0, Iter = 47, Block = 0, Slice = 0
[INFO] [2015-11-26 19:21:49] Rank = 0, Alias Time used: 1.49 s 
[INFO] [2015-11-26 19:22:13] Rank = 0, Training Time used: 140.08 s 
[INFO] [2015-11-26 19:22:13] Rank = 0, sampling throughput: 118335.397796 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:22:13] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:22:14] Rank = 0, Iter = 48, Block = 0, Slice = 0
[INFO] [2015-11-26 19:22:14] Rank = 0, Alias Time used: 1.68 s 
[INFO] [2015-11-26 19:22:38] Rank = 0, Training Time used: 138.16 s 
[INFO] [2015-11-26 19:22:38] Rank = 0, sampling throughput: 119980.225881 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:22:40] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:22:41] Rank = 0, Iter = 49, Block = 0, Slice = 0
[INFO] [2015-11-26 19:22:41] Rank = 0, Alias Time used: 1.54 s 
[INFO] [2015-11-26 19:23:07] Rank = 0, Training Time used: 152.84 s 
[INFO] [2015-11-26 19:23:07] Rank = 0, sampling throughput: 108459.268085 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:23:07] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:23:08] Rank = 0, Iter = 50, Block = 0, Slice = 0
[INFO] [2015-11-26 19:23:08] Rank = 0, Alias Time used: 1.40 s 
[INFO] [2015-11-26 19:23:30] Rank = 0, Training Time used: 133.89 s 
[INFO] [2015-11-26 19:23:30] Rank = 0, sampling throughput: 123807.136131 (tokens/thread/sec) 
[INFO] [2015-11-26 19:23:40] doc likelihood : -4.314949e+08
[INFO] [2015-11-26 19:23:40] word likelihood : 8.308841e+08
[INFO] [2015-11-26 19:23:40] Normalized likelihood : -1.584257e+09
[INFO] [2015-11-26 19:23:40] Rank = 0, Evaluation Time used: 41.05 s 
[DEBUG] [2015-11-26 19:23:40] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:23:41] Rank = 0, Iter = 51, Block = 0, Slice = 0
[INFO] [2015-11-26 19:23:41] Rank = 0, Alias Time used: 1.69 s 
[INFO] [2015-11-26 19:24:12] Rank = 0, Training Time used: 154.38 s 
[INFO] [2015-11-26 19:24:12] Rank = 0, sampling throughput: 107378.348014 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:24:12] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:24:13] Rank = 0, Iter = 52, Block = 0, Slice = 0
[INFO] [2015-11-26 19:24:13] Rank = 0, Alias Time used: 1.37 s 
[INFO] [2015-11-26 19:24:36] Rank = 0, Training Time used: 139.34 s 
[INFO] [2015-11-26 19:24:36] Rank = 0, sampling throughput: 118964.549555 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:24:38] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:24:38] Rank = 0, Iter = 53, Block = 0, Slice = 0
[INFO] [2015-11-26 19:24:39] Rank = 0, Alias Time used: 2.05 s 
[INFO] [2015-11-26 19:25:04] Rank = 0, Training Time used: 142.71 s 
[INFO] [2015-11-26 19:25:04] Rank = 0, sampling throughput: 116158.505032 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:25:04] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:25:05] Rank = 0, Iter = 54, Block = 0, Slice = 0
[INFO] [2015-11-26 19:25:05] Rank = 0, Alias Time used: 1.50 s 
[INFO] [2015-11-26 19:25:27] Rank = 0, Training Time used: 131.35 s 
[INFO] [2015-11-26 19:25:27] Rank = 0, sampling throughput: 126155.166989 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:25:30] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:25:31] Rank = 0, Iter = 55, Block = 0, Slice = 0
[INFO] [2015-11-26 19:25:31] Rank = 0, Alias Time used: 1.65 s 
[INFO] [2015-11-26 19:25:55] Rank = 0, Training Time used: 143.52 s 
[INFO] [2015-11-26 19:25:55] Rank = 0, sampling throughput: 115439.869846 (tokens/thread/sec) 
[INFO] [2015-11-26 19:26:01] doc likelihood : -4.280392e+08
[INFO] [2015-11-26 19:26:01] word likelihood : 8.438210e+08
[INFO] [2015-11-26 19:26:01] Normalized likelihood : -1.583670e+09
[INFO] [2015-11-26 19:26:01] Rank = 0, Evaluation Time used: 26.59 s 
[DEBUG] [2015-11-26 19:26:01] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:26:02] Rank = 0, Iter = 56, Block = 0, Slice = 0
[INFO] [2015-11-26 19:26:02] Rank = 0, Alias Time used: 1.49 s 
[INFO] [2015-11-26 19:26:24] Rank = 0, Training Time used: 125.45 s 
[INFO] [2015-11-26 19:26:24] Rank = 0, sampling throughput: 132138.935103 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:26:28] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:26:29] Rank = 0, Iter = 57, Block = 0, Slice = 0
[INFO] [2015-11-26 19:26:29] Rank = 0, Alias Time used: 1.37 s 
[INFO] [2015-11-26 19:26:51] Rank = 0, Training Time used: 129.02 s 
[INFO] [2015-11-26 19:26:51] Rank = 0, sampling throughput: 128398.332540 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:26:53] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:26:54] Rank = 0, Iter = 58, Block = 0, Slice = 0
[INFO] [2015-11-26 19:26:54] Rank = 0, Alias Time used: 1.39 s 
[INFO] [2015-11-26 19:27:18] Rank = 0, Training Time used: 139.39 s 
[INFO] [2015-11-26 19:27:18] Rank = 0, sampling throughput: 118926.323367 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:27:19] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:27:19] Rank = 0, Iter = 59, Block = 0, Slice = 0
[INFO] [2015-11-26 19:27:20] Rank = 0, Alias Time used: 1.57 s 
[INFO] [2015-11-26 19:27:45] Rank = 0, Training Time used: 141.16 s 
[INFO] [2015-11-26 19:27:45] Rank = 0, sampling throughput: 117430.336350 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:27:45] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:27:46] Rank = 0, Iter = 60, Block = 0, Slice = 0
[INFO] [2015-11-26 19:27:46] Rank = 0, Alias Time used: 1.62 s 
[INFO] [2015-11-26 19:28:09] Rank = 0, Training Time used: 133.56 s 
[INFO] [2015-11-26 19:28:09] Rank = 0, sampling throughput: 124116.618109 (tokens/thread/sec) 
[INFO] [2015-11-26 19:28:15] doc likelihood : -4.257501e+08
[INFO] [2015-11-26 19:28:15] word likelihood : 8.539954e+08
[INFO] [2015-11-26 19:28:15] Normalized likelihood : -1.583021e+09
[INFO] [2015-11-26 19:28:15] Rank = 0, Evaluation Time used: 34.62 s 
[DEBUG] [2015-11-26 19:28:15] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:28:16] Rank = 0, Iter = 61, Block = 0, Slice = 0
[INFO] [2015-11-26 19:28:16] Rank = 0, Alias Time used: 1.44 s 
[INFO] [2015-11-26 19:28:36] Rank = 0, Training Time used: 107.46 s 
[INFO] [2015-11-26 19:28:36] Rank = 0, sampling throughput: 154255.024397 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:28:46] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:28:47] Rank = 0, Iter = 62, Block = 0, Slice = 0
[INFO] [2015-11-26 19:28:47] Rank = 0, Alias Time used: 1.75 s 
[INFO] [2015-11-26 19:29:08] Rank = 0, Training Time used: 128.39 s 
[INFO] [2015-11-26 19:29:08] Rank = 0, sampling throughput: 129116.484644 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:29:11] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:29:12] Rank = 0, Iter = 63, Block = 0, Slice = 0
[INFO] [2015-11-26 19:29:12] Rank = 0, Alias Time used: 1.43 s 
[INFO] [2015-11-26 19:29:35] Rank = 0, Training Time used: 136.27 s 
[INFO] [2015-11-26 19:29:35] Rank = 0, sampling throughput: 121650.262386 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:29:36] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:29:37] Rank = 0, Iter = 64, Block = 0, Slice = 0
[INFO] [2015-11-26 19:29:37] Rank = 0, Alias Time used: 1.44 s 
[INFO] [2015-11-26 19:30:01] Rank = 0, Training Time used: 137.20 s 
[INFO] [2015-11-26 19:30:01] Rank = 0, sampling throughput: 120822.632376 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:30:02] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:30:02] Rank = 0, Iter = 65, Block = 0, Slice = 0
[INFO] [2015-11-26 19:30:03] Rank = 0, Alias Time used: 1.38 s 
[INFO] [2015-11-26 19:30:30] Rank = 0, Training Time used: 150.12 s 
[INFO] [2015-11-26 19:30:30] Rank = 0, sampling throughput: 110424.012822 (tokens/thread/sec) 
[INFO] [2015-11-26 19:30:33] doc likelihood : -4.240845e+08
[INFO] [2015-11-26 19:30:33] word likelihood : 8.621267e+08
[INFO] [2015-11-26 19:30:33] Normalized likelihood : -1.582389e+09
[INFO] [2015-11-26 19:30:33] Rank = 0, Evaluation Time used: 18.53 s 
[DEBUG] [2015-11-26 19:30:33] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:30:34] Rank = 0, Iter = 66, Block = 0, Slice = 0
[INFO] [2015-11-26 19:30:34] Rank = 0, Alias Time used: 1.66 s 
[INFO] [2015-11-26 19:30:54] Rank = 0, Training Time used: 122.02 s 
[INFO] [2015-11-26 19:30:54] Rank = 0, sampling throughput: 135676.916831 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:31:01] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:31:01] Rank = 0, Iter = 67, Block = 0, Slice = 0
[INFO] [2015-11-26 19:31:02] Rank = 0, Alias Time used: 1.50 s 
[INFO] [2015-11-26 19:31:28] Rank = 0, Training Time used: 138.75 s 
[INFO] [2015-11-26 19:31:28] Rank = 0, sampling throughput: 119397.508518 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:31:30] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:31:31] Rank = 0, Iter = 68, Block = 0, Slice = 0
[INFO] [2015-11-26 19:31:31] Rank = 0, Alias Time used: 1.46 s 
[INFO] [2015-11-26 19:31:52] Rank = 0, Training Time used: 123.99 s 
[INFO] [2015-11-26 19:31:52] Rank = 0, sampling throughput: 133694.848980 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:31:57] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:31:57] Rank = 0, Iter = 69, Block = 0, Slice = 0
[INFO] [2015-11-26 19:31:58] Rank = 0, Alias Time used: 1.67 s 
[INFO] [2015-11-26 19:32:21] Rank = 0, Training Time used: 138.28 s 
[INFO] [2015-11-26 19:32:21] Rank = 0, sampling throughput: 119878.573570 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:32:22] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:32:23] Rank = 0, Iter = 70, Block = 0, Slice = 0
[INFO] [2015-11-26 19:32:23] Rank = 0, Alias Time used: 1.35 s 
[INFO] [2015-11-26 19:32:40] Rank = 0, Training Time used: 103.31 s 
[INFO] [2015-11-26 19:32:40] Rank = 0, sampling throughput: 160461.523622 (tokens/thread/sec) 
[INFO] [2015-11-26 19:32:51] doc likelihood : -4.229418e+08
[INFO] [2015-11-26 19:32:52] word likelihood : 8.688454e+08
[INFO] [2015-11-26 19:32:52] Normalized likelihood : -1.581791e+09
[INFO] [2015-11-26 19:32:52] Rank = 0, Evaluation Time used: 64.24 s 
[DEBUG] [2015-11-26 19:32:52] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:32:53] Rank = 0, Iter = 71, Block = 0, Slice = 0
[INFO] [2015-11-26 19:32:53] Rank = 0, Alias Time used: 1.71 s 
[INFO] [2015-11-26 19:33:13] Rank = 0, Training Time used: 119.84 s 
[INFO] [2015-11-26 19:33:13] Rank = 0, sampling throughput: 138323.524062 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:33:16] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:33:17] Rank = 0, Iter = 72, Block = 0, Slice = 0
[INFO] [2015-11-26 19:33:17] Rank = 0, Alias Time used: 1.39 s 
[INFO] [2015-11-26 19:33:39] Rank = 0, Training Time used: 129.61 s 
[INFO] [2015-11-26 19:33:39] Rank = 0, sampling throughput: 127893.766193 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:33:42] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:33:42] Rank = 0, Iter = 73, Block = 0, Slice = 0
[INFO] [2015-11-26 19:33:43] Rank = 0, Alias Time used: 1.69 s 
[INFO] [2015-11-26 19:34:04] Rank = 0, Training Time used: 126.04 s 
[INFO] [2015-11-26 19:34:04] Rank = 0, sampling throughput: 131522.060527 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:34:08] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:34:09] Rank = 0, Iter = 74, Block = 0, Slice = 0
[INFO] [2015-11-26 19:34:09] Rank = 0, Alias Time used: 1.56 s 
[INFO] [2015-11-26 19:34:29] Rank = 0, Training Time used: 119.93 s 
[INFO] [2015-11-26 19:34:29] Rank = 0, sampling throughput: 138219.504981 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:34:33] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:34:34] Rank = 0, Iter = 75, Block = 0, Slice = 0
[INFO] [2015-11-26 19:34:34] Rank = 0, Alias Time used: 1.30 s 
[INFO] [2015-11-26 19:34:59] Rank = 0, Training Time used: 144.57 s 
[INFO] [2015-11-26 19:34:59] Rank = 0, sampling throughput: 114662.933687 (tokens/thread/sec) 
[INFO] [2015-11-26 19:35:02] doc likelihood : -4.220603e+08
[INFO] [2015-11-26 19:35:02] word likelihood : 8.744806e+08
[INFO] [2015-11-26 19:35:02] Normalized likelihood : -1.581272e+09
[INFO] [2015-11-26 19:35:02] Rank = 0, Evaluation Time used: 16.66 s 
[DEBUG] [2015-11-26 19:35:02] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:35:03] Rank = 0, Iter = 76, Block = 0, Slice = 0
[INFO] [2015-11-26 19:35:03] Rank = 0, Alias Time used: 1.30 s 
[INFO] [2015-11-26 19:35:24] Rank = 0, Training Time used: 123.81 s 
[INFO] [2015-11-26 19:35:24] Rank = 0, sampling throughput: 133895.158290 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:35:28] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:35:29] Rank = 0, Iter = 77, Block = 0, Slice = 0
[INFO] [2015-11-26 19:35:29] Rank = 0, Alias Time used: 1.77 s 
[INFO] [2015-11-26 19:35:52] Rank = 0, Training Time used: 126.83 s 
[INFO] [2015-11-26 19:35:52] Rank = 0, sampling throughput: 130701.121912 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:35:56] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:35:57] Rank = 0, Iter = 78, Block = 0, Slice = 0
[INFO] [2015-11-26 19:35:57] Rank = 0, Alias Time used: 1.56 s 
[INFO] [2015-11-26 19:36:19] Rank = 0, Training Time used: 123.26 s 
[INFO] [2015-11-26 19:36:19] Rank = 0, sampling throughput: 134399.596618 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:36:24] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:36:25] Rank = 0, Iter = 79, Block = 0, Slice = 0
[INFO] [2015-11-26 19:36:25] Rank = 0, Alias Time used: 1.23 s 
[INFO] [2015-11-26 19:36:47] Rank = 0, Training Time used: 133.95 s 
[INFO] [2015-11-26 19:36:47] Rank = 0, sampling throughput: 123757.576251 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:36:48] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:36:49] Rank = 0, Iter = 80, Block = 0, Slice = 0
[INFO] [2015-11-26 19:36:49] Rank = 0, Alias Time used: 1.49 s 
[INFO] [2015-11-26 19:37:11] Rank = 0, Training Time used: 133.50 s 
[INFO] [2015-11-26 19:37:11] Rank = 0, sampling throughput: 124170.388139 (tokens/thread/sec) 
[INFO] [2015-11-26 19:37:16] doc likelihood : -4.215044e+08
[INFO] [2015-11-26 19:37:17] word likelihood : 8.793446e+08
[INFO] [2015-11-26 19:37:17] Normalized likelihood : -1.580807e+09
[INFO] [2015-11-26 19:37:17] Rank = 0, Evaluation Time used: 25.43 s 
[DEBUG] [2015-11-26 19:37:17] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:37:17] Rank = 0, Iter = 81, Block = 0, Slice = 0
[INFO] [2015-11-26 19:37:18] Rank = 0, Alias Time used: 1.43 s 
[INFO] [2015-11-26 19:37:45] Rank = 0, Training Time used: 146.71 s 
[INFO] [2015-11-26 19:37:45] Rank = 0, sampling throughput: 112989.725378 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:37:45] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:37:46] Rank = 0, Iter = 82, Block = 0, Slice = 0
[INFO] [2015-11-26 19:37:46] Rank = 0, Alias Time used: 1.24 s 
[INFO] [2015-11-26 19:38:08] Rank = 0, Training Time used: 130.01 s 
[INFO] [2015-11-26 19:38:08] Rank = 0, sampling throughput: 127508.318693 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:38:10] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:38:10] Rank = 0, Iter = 83, Block = 0, Slice = 0
[INFO] [2015-11-26 19:38:11] Rank = 0, Alias Time used: 1.55 s 
[INFO] [2015-11-26 19:38:36] Rank = 0, Training Time used: 141.07 s 
[INFO] [2015-11-26 19:38:36] Rank = 0, sampling throughput: 117505.630428 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:38:36] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:38:37] Rank = 0, Iter = 84, Block = 0, Slice = 0
[INFO] [2015-11-26 19:38:37] Rank = 0, Alias Time used: 1.47 s 
[INFO] [2015-11-26 19:38:56] Rank = 0, Training Time used: 110.34 s 
[INFO] [2015-11-26 19:38:56] Rank = 0, sampling throughput: 150232.479256 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:39:01] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:39:02] Rank = 0, Iter = 85, Block = 0, Slice = 0
[INFO] [2015-11-26 19:39:02] Rank = 0, Alias Time used: 1.58 s 
[INFO] [2015-11-26 19:39:26] Rank = 0, Training Time used: 137.54 s 
[INFO] [2015-11-26 19:39:26] Rank = 0, sampling throughput: 120524.656035 (tokens/thread/sec) 
[INFO] [2015-11-26 19:39:31] doc likelihood : -4.210534e+08
[INFO] [2015-11-26 19:39:32] word likelihood : 8.835320e+08
[INFO] [2015-11-26 19:39:32] Normalized likelihood : -1.580384e+09
[INFO] [2015-11-26 19:39:32] Rank = 0, Evaluation Time used: 23.83 s 
[DEBUG] [2015-11-26 19:39:32] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:39:32] Rank = 0, Iter = 86, Block = 0, Slice = 0
[INFO] [2015-11-26 19:39:33] Rank = 0, Alias Time used: 1.56 s 
[INFO] [2015-11-26 19:39:58] Rank = 0, Training Time used: 138.72 s 
[INFO] [2015-11-26 19:39:58] Rank = 0, sampling throughput: 119450.538189 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:39:58] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:39:59] Rank = 0, Iter = 87, Block = 0, Slice = 0
[INFO] [2015-11-26 19:39:59] Rank = 0, Alias Time used: 1.44 s 
[INFO] [2015-11-26 19:40:19] Rank = 0, Training Time used: 117.04 s 
[INFO] [2015-11-26 19:40:19] Rank = 0, sampling throughput: 141633.889817 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:40:22] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:40:23] Rank = 0, Iter = 88, Block = 0, Slice = 0
[INFO] [2015-11-26 19:40:24] Rank = 0, Alias Time used: 2.08 s 
[INFO] [2015-11-26 19:40:45] Rank = 0, Training Time used: 119.46 s 
[INFO] [2015-11-26 19:40:45] Rank = 0, sampling throughput: 138661.819506 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:40:50] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:40:51] Rank = 0, Iter = 89, Block = 0, Slice = 0
[INFO] [2015-11-26 19:40:51] Rank = 0, Alias Time used: 1.16 s 
[INFO] [2015-11-26 19:41:09] Rank = 0, Training Time used: 110.17 s 
[INFO] [2015-11-26 19:41:09] Rank = 0, sampling throughput: 150464.912253 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:41:16] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:41:17] Rank = 0, Iter = 90, Block = 0, Slice = 0
[INFO] [2015-11-26 19:41:17] Rank = 0, Alias Time used: 1.59 s 
[INFO] [2015-11-26 19:41:40] Rank = 0, Training Time used: 133.89 s 
[INFO] [2015-11-26 19:41:40] Rank = 0, sampling throughput: 123735.482634 (tokens/thread/sec) 
[INFO] [2015-11-26 19:41:46] doc likelihood : -4.206959e+08
[INFO] [2015-11-26 19:41:46] word likelihood : 8.872202e+08
[INFO] [2015-11-26 19:41:46] Normalized likelihood : -1.580009e+09
[INFO] [2015-11-26 19:41:46] Rank = 0, Evaluation Time used: 31.54 s 
[DEBUG] [2015-11-26 19:41:46] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:41:47] Rank = 0, Iter = 91, Block = 0, Slice = 0
[INFO] [2015-11-26 19:41:47] Rank = 0, Alias Time used: 1.69 s 
[INFO] [2015-11-26 19:42:11] Rank = 0, Training Time used: 137.06 s 
[INFO] [2015-11-26 19:42:11] Rank = 0, sampling throughput: 120949.302820 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:42:13] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:42:14] Rank = 0, Iter = 92, Block = 0, Slice = 0
[INFO] [2015-11-26 19:42:14] Rank = 0, Alias Time used: 1.49 s 
[INFO] [2015-11-26 19:42:36] Rank = 0, Training Time used: 129.69 s 
[INFO] [2015-11-26 19:42:36] Rank = 0, sampling throughput: 127816.967768 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:42:37] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:42:38] Rank = 0, Iter = 93, Block = 0, Slice = 0
[INFO] [2015-11-26 19:42:38] Rank = 0, Alias Time used: 1.43 s 
[INFO] [2015-11-26 19:43:01] Rank = 0, Training Time used: 134.25 s 
[INFO] [2015-11-26 19:43:01] Rank = 0, sampling throughput: 123474.526856 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:43:01] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:43:02] Rank = 0, Iter = 94, Block = 0, Slice = 0
[INFO] [2015-11-26 19:43:02] Rank = 0, Alias Time used: 1.37 s 
[INFO] [2015-11-26 19:43:24] Rank = 0, Training Time used: 129.88 s 
[INFO] [2015-11-26 19:43:24] Rank = 0, sampling throughput: 127633.292251 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:43:25] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:43:26] Rank = 0, Iter = 95, Block = 0, Slice = 0
[INFO] [2015-11-26 19:43:26] Rank = 0, Alias Time used: 1.39 s 
[INFO] [2015-11-26 19:43:43] Rank = 0, Training Time used: 99.65 s 
[INFO] [2015-11-26 19:43:43] Rank = 0, sampling throughput: 166217.845728 (tokens/thread/sec) 
[INFO] [2015-11-26 19:43:53] doc likelihood : -4.205036e+08
[INFO] [2015-11-26 19:43:54] word likelihood : 8.904672e+08
[INFO] [2015-11-26 19:43:54] Normalized likelihood : -1.579677e+09
[INFO] [2015-11-26 19:43:54] Rank = 0, Evaluation Time used: 55.21 s 
[DEBUG] [2015-11-26 19:43:54] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:43:54] Rank = 0, Iter = 96, Block = 0, Slice = 0
[INFO] [2015-11-26 19:43:55] Rank = 0, Alias Time used: 1.38 s 
[INFO] [2015-11-26 19:44:16] Rank = 0, Training Time used: 125.78 s 
[INFO] [2015-11-26 19:44:16] Rank = 0, sampling throughput: 131736.899645 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:44:18] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:44:19] Rank = 0, Iter = 97, Block = 0, Slice = 0
[INFO] [2015-11-26 19:44:19] Rank = 0, Alias Time used: 1.30 s 
[INFO] [2015-11-26 19:44:36] Rank = 0, Training Time used: 102.23 s 
[INFO] [2015-11-26 19:44:36] Rank = 0, sampling throughput: 162153.926892 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:44:42] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:44:42] Rank = 0, Iter = 98, Block = 0, Slice = 0
[INFO] [2015-11-26 19:44:43] Rank = 0, Alias Time used: 1.47 s 
[INFO] [2015-11-26 19:45:12] Rank = 0, Training Time used: 147.84 s 
[INFO] [2015-11-26 19:45:12] Rank = 0, sampling throughput: 112124.181078 (tokens/thread/sec) 
[DEBUG] [2015-11-26 19:45:13] Request params. start = 0, end = 101635
[INFO] [2015-11-26 19:45:13] Rank = 0, Iter = 99, Block = 0, Slice = 0
[INFO] [2015-11-26 19:45:14] Rank = 0, Alias Time used: 1.69 s 
[INFO] [2015-11-26 19:45:38] Rank = 0, Training Time used: 138.22 s 
[INFO] [2015-11-26 19:45:38] Rank = 0, sampling throughput: 119927.936166 (tokens/thread/sec) 
[INFO] [2015-11-26 19:45:39] Rank 0/1: End of training.
[INFO] [2015-11-26 19:45:39] Server 0: Received close message from worker 0.
[INFO] [2015-11-26 19:45:39] Server 0: Dump model...
[INFO] [2015-11-26 19:45:41] Server 0 closed.
[INFO] [2015-11-26 19:45:42] Rank 0/1: Multiverso closed successfully.

Thank you.

Marco

feiga commented 8 years ago

Hi, Marco.

Sorry the log is a little confusing. But it converges. Maybe discussion in this issue can help.

ross85 commented 8 years ago

Hi, thank you for your answer.