Implementation and experimentation on temporal negative sampling strategies

[VaghehDashti]

This issue will be for various temporal negative sampling strategies. The first temporal negative sampling that I will be working on is as follows: When training the neural models on year t, we are going to use the unigram distribution of year t - k to generate negative samples. Also for the first k years, the model will not utilize negative sampling in training.

[VaghehDashti]

Hi @hosseinfani, Here's an update regarding our first temporal negative sampling strategy: Just a reminder that in this first method when training the model on year [t], we are going to use the unigram distribution of year [t - k] to generate negative samples.

The first results are for k=1 and they are not good unfortunately. They all have a much lower performance on all metrics. I will run on k=5 and k=10 to see if it gets better or worse. Also, the github results are not complete due to some minor issues, I will update them asap. dblp:

  | P_2 | P_5 | P_10 | r_2 | r_5 | r_10 | ndcg_2 | ndcg_5 | ndcg_10 | map_2 | map_5 | map_10 | aucroc -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- rrn | 0.0570 | 0.0391 | 0.0472 | 0.0380 | 0.0630 | 0.1552 | 0.0478 | 0.0523 | 0.0959 | 0.0217 | 0.0281 | 0.0446 | 0.5073 tbnn | 0.1189 | 0.1413 | 0.1664 | 0.0706 | 0.2090 | 0.4984 | 0.1126 | 0.1689 | 0.3031 | 0.0484 | 0.0845 | 0.1223 | 0.7308 tbnn_emb | 0.2996 | 0.2938 | 0.2811 | 0.1816 | 0.4433 | 0.8431 | 0.3048 | 0.3860 | 0.5721 | 0.1411 | 0.2095 | 0.2635 | 0.7483 tbnn_dt2v_emb | 0.4299 | 0.3973 | 0.3612 | 0.2601 | 0.5963 | 1.0801 | 0.4284 | 0.5221 | 0.7465 | 0.1947 | 0.2864 | 0.3520 | 0.7701 tbnn_tu1 | 0.0358 | 0.0358 | 0.0319 | 0.0212 | 0.0549 | 0.0989 | 0.0336 | 0.0452 | 0.0645 | 0.0146 | 0.0231 | 0.0285 | 0.517676 tbnn_emb_tu1 | 0.0342 | 0.0202 | 0.0228 | 0.0207 | 0.0305 | 0.0706 | 0.0301 | 0.0277 | 0.046 | 0.0127 | 0.0151 | 0.0203 | 0.538863 tbnn_dt2v_emb_tu1 | 0.013 | 0.0156 | 0.0182 | 7.98E-03 | 0.0245 | 0.0574 | 0.0116 | 0.0183 | 0.0334 | 4.94E-03 | 8.85E-03 | 0.0132 | 0.544872

imdb:

  | P_2 | P_5 | P_10 | r_2 | r_5 | r_10 | ndcg_2 | ndcg_5 | ndcg_10 | map_2 | map_5 | map_10 | aucroc -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- rrn | 0 | 0.8511 | 0.8511 | 0 | 1.4184 | 2.8369 | 0.0000 | 0.8163 | 1.4606 | 0.0000 | 0.3191 | 0.6265 | 0.5222 tbnn | 0.8511 | 1.5319 | 1.4043 | 0.5319 | 2.4610 | 4.4965 | 0.7548 | 1.7381 | 2.6829 | 0.3369 | 0.8215 | 1.1674 | 0.6346 tbnn_emb | 0.8511 | 1.1064 | 1.0638 | 0.5674 | 1.7518 | 1.3262 | 0.9474 | 1.4848 | 2.2007 | 0.4965 | 0.8138 | 1.0099 | 0.6687 tbnn_dt2v_emb | 1.9149 | 1.1915 | 1.4468 | 1.2411 | 1.9504 | 4.5532 | 1.8667 | 1.8703 | 3.0303 | 0.9043 | 1.1099 | 1.4293 | 0.6656 tbnn_tu1 | 0.4255 | 0.766 | 0.6809 | 0.2837 | 1.2766 | 2.2695 | 0.4255 | 0.9204 | 1.3876 | 0.2128 | 0.4563 | 0.6376 | 0.461106 tbnn_emb_tu1 | 0.4255 | 0.5957 | 0.5957 | 0.227 | 0.8794 | 1.8156 | 0.5218 | 0.7691 | 1.1911 | 0.227 | 0.4066 | 0.5466 | 0.475228 tbnn_dt2v_emb_tu1 | 0 | 0.5106 | 0.6383 | 0 | 0.7943 | 2.0709 | 0 | 0.4973 | 1.0861 | 0 | 0.2033 | 0.3789 | 0.460456

uspt:

  | P_2 | P_5 | P_10 | r_2 | r_5 | r_10 | ndcg_2 | ndcg_5 | ndcg_10 | map_2 | map_5 | map_10 | aucroc -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- rrn | 0.0239 | 0.0383 | 0.0654 | 0.0140 | 0.0500 | 0.1370 | 0.0221 | 0.0408 | 0.0868 | 0.0096 | 0.0186 | 0.0340 | 0.5160 tbnn | 0.1843 | 0.1841 | 0.2029 | 0.0933 | 0.2321 | 0.5158 | 0.1794 | 0.2152 | 0.3481 | 0.0681 | 0.1056 | 0.1429 | 0.7544 tbnn_emb | 0.8272 | 0.7539 | 0.7042 | 0.3970 | 0.9021 | 1.6933 | 0.8457 | 0.9057 | 1.2657 | 0.3104 | 0.4533 | 0.5679 | 0.8359 tbnn_dt2v_emb | 1.2268 | 1.0583 | 0.9324 | 0.6037 | 1.2928 | 2.2518 | 1.2322 | 1.2960 | 1.7348 | 0.4626 | 0.6659 | 0.8118 | 0.8534 tbnn_tu1 | 0.0223 | 0.0239 | 0.0241 | 0.0119 | 0.0324 | 0.0668 | 0.0216 | 0.0286 | 0.0441 | 8.45E-03 | 0.0139 | 0.0184 | 0.496715 tbnn_emb_tu1 | 0.0622 | 0.0721 | 0.0777 | 0.0258 | 0.0804 | 0.168 | 0.0629 | 0.0804 | 0.1182 | 0.0189 | 0.033 | 0.0455 | 0.58621 tbnn_dt2v_emb_tu1 | 0.067 | 0.0756 | 0.0767 | 0.0239 | 0.07 | 0.1479 | 0.0695 | 0.0852 | 0.1136 | 0.02 | 0.0324 | 0.0439 | 0.603763

gith:

  | P_2 | P_5 | P_10 | r_2 | r_5 | r_10 | ndcg_2 | ndcg_5 | ndcg_10 | map_2 | map_5 | map_10 | aucroc -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- rrn |   |   |   |   |   |   |   |   |   |   |   |   |   tbnn | 0.0178 | 0.0182 | 0.0186 | 0.0078 | 0.0161 | 0.0294 | 0.0183 | 0.0205 | 0.0264 | 0.0063 | 0.0093 | 0.0126 | 0.5234 tbnn_emb | 0.0149 | 0.0154 | 0.0143 | 0.0069 | 0.0158 | 0.0259 | 0.0160 | 0.0183 | 0.0230 | 0.0058 | 0.0087 | 0.0110 | 0.5936 tbnn_dt2v_emb |   |   |   |   |   |   |   |   |   |   |   |   |   tbnn_tu1 | 0.013861 | 0.013861 | 0.016436 | 0.003555 | 0.008322 | 0.019873 | 0.013861 | 0.015084 | 0.019793 | 0.003047 | 0.004508 | 0.006927 | 0.532643 tbnn_emb_tu1 | 0.016832 | 0.020198 | 0.018614 | 0.005275 | 0.012293 | 0.023821 | 0.017056 | 0.020098 | 0.023306 | 0.004022 | 0.006585 | 0.008941 | 0.583891 tbnn_dt2v_emb_tu1 | 0.008911 | 0.007921 | 0.010693 | 0.003719 | 0.007194 | 0.019331 | 0.008239 | 0.008746 | 0.01491 | 0.002949 | 0.003968 | 0.006275 | 0.575736

[VaghehDashti]

Hi @hosseinfani, Here is the final results of experiments on the first temporal negative sampling strategy. As can be seen, unfortunately, the first temporal ns strategy decreases performance with different ks [1,5,10]. I have implemented the second temporal ns, where instead of just using year [t-k] to compute the unigram, we use data from all year up to year [t-k]. I will be running the experiments on k=[1,5,10] and update here when they are ready.

dblp:

  | P_2 | P_5 | P_10 | r_2 | r_5 | r_10 | ndcg_2 | ndcg_5 | ndcg_10 | map_2 | map_5 | map_10 | aucroc -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- rrn | 0.0570 | 0.0391 | 0.0472 | 0.0380 | 0.0630 | 0.1552 | 0.0478 | 0.0523 | 0.0959 | 0.0217 | 0.0281 | 0.0446 | 0.5073 tbnn | 0.1189 | 0.1413 | 0.1664 | 0.0706 | 0.2090 | 0.4984 | 0.1126 | 0.1689 | 0.3031 | 0.0484 | 0.0845 | 0.1223 | 0.7308 tbnn_emb | 0.2996 | 0.2938 | 0.2811 | 0.1816 | 0.4433 | 0.8431 | 0.3048 | 0.3860 | 0.5721 | 0.1411 | 0.2095 | 0.2635 | 0.7483 tbnn_dt2v_emb | 0.4299 | 0.3973 | 0.3612 | 0.2601 | 0.5963 | 1.0801 | 0.4284 | 0.5221 | 0.7465 | 0.1947 | 0.2864 | 0.3520 | 0.7701 tbnn_tu1 | 0.0358 | 0.0358 | 0.0319 | 0.0212 | 0.0549 | 0.0989 | 0.0336 | 0.0452 | 0.0645 | 0.0146 | 0.0231 | 0.0285 | 0.517676 tbnn_emb_tu1 | 0.0342 | 0.0202 | 0.0228 | 0.0207 | 0.0305 | 0.0706 | 0.0301 | 0.0277 | 0.046 | 0.0127 | 0.0151 | 0.0203 | 0.538863 tbnn_dt2v_emb_tu1 | 0.013 | 0.0156 | 0.0182 | 7.98E-03 | 0.0245 | 0.0574 | 0.0116 | 0.0183 | 0.0334 | 4.94E-03 | 8.85E-03 | 0.0132 | 0.544872 tbnn_tu5 | 0.0537 | 0.0469 | 0.0378 | 0.0332 | 0.0739 | 0.1182 | 0.0497 | 0.0619 | 0.0823 | 0.0220 | 0.0329 | 0.0387 | 0.573205 tbnn_emb_tu5 | 0.0342 | 0.0391 | 0.0430 | 0.0215 | 0.0599 | 0.1321 | 0.0368 | 0.0512 | 0.0849 | 0.0180 | 0.0277 | 0.0373 | 0.582192 tbnn_dt2v_emb_tu5 | 0.0163 | 0.0208 | 0.0257 | 0.0102 | 0.0332 | 0.0794 | 0.0170 | 0.0265 | 0.0480 | 0.0081 | 0.0138 | 0.0199 | 0.594107 tbnn_tu10 | 0.0635 | 0.0567 | 0.0492 | 0.0378 | 0.0848 | 0.1498 | 0.0646 | 0.0767 | 0.1057 | 0.0296 | 0.0419 | 0.0501 | 0.6038 tbnn_emb_tu10 | 0.0358 | 0.0436 | 0.0531 | 0.0226 | 0.0663 | 0.1627 | 0.0388 | 0.0566 | 0.1009 | 0.0188 | 0.0304 | 0.0433 | 0.6168 tbnn_dt2v_emb_tu10 | 0.0309 | 0.0280 | 0.0410 | 0.0193 | 0.0442 | 0.1283 | 0.0313 | 0.0382 | 0.0767 | 0.0147 | 0.0208 | 0.0321 | 0.6308

imdb:

  | P_2 | P_5 | P_10 | r_2 | r_5 | r_10 | ndcg_2 | ndcg_5 | ndcg_10 | map_2 | map_5 | map_10 | aucroc -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- rrn | 0 | 0.8511 | 0.8511 | 0 | 1.4184 | 2.8369 | 0.0000 | 0.8163 | 1.4606 | 0.0000 | 0.3191 | 0.6265 | 0.5222 tbnn | 0.8511 | 1.5319 | 1.4043 | 0.5319 | 2.4610 | 4.4965 | 0.7548 | 1.7381 | 2.6829 | 0.3369 | 0.8215 | 1.1674 | 0.6346 tbnn_emb | 0.8511 | 1.1064 | 1.0638 | 0.5674 | 1.7518 | 1.3262 | 0.9474 | 1.4848 | 2.2007 | 0.4965 | 0.8138 | 1.0099 | 0.6687 tbnn_dt2v_emb | 1.9149 | 1.1915 | 1.4468 | 1.2411 | 1.9504 | 4.5532 | 1.8667 | 1.8703 | 3.0303 | 0.9043 | 1.1099 | 1.4293 | 0.6656 tbnn_tu1 | 0.4255 | 0.766 | 0.6809 | 0.2837 | 1.2766 | 2.2695 | 0.4255 | 0.9204 | 1.3876 | 0.2128 | 0.4563 | 0.6376 | 0.461106 tbnn_emb_tu1 | 0.4255 | 0.5957 | 0.5957 | 0.227 | 0.8794 | 1.8156 | 0.5218 | 0.7691 | 1.1911 | 0.227 | 0.4066 | 0.5466 | 0.475228 tbnn_dt2v_emb_tu1 | 0 | 0.5106 | 0.6383 | 0 | 0.7943 | 2.0709 | 0 | 0.4973 | 1.0861 | 0 | 0.2033 | 0.3789 | 0.460456 tbnn_tu5 | 0 | 0.3404 | 0.383 | 0 | 0.5674 | 1.2766 | 0 | 0.3353 | 0.6597 | 0 | 0.1348 | 0.2304 | 0.42074 tbnn_emb_tu5 | 0.6383 | 0.5106 | 0.4681 | 0.3688 | 0.7943 | 1.5035 | 0.5902 | 0.666 | 0.9845 | 0.2553 | 0.3546 | 0.4462 | 0.398292 tbnn_dt2v_emb_tu5 | 0.8511 | 0.766 | 0.5532 | 0.5674 | 1.2766 | 1.7872 | 0.8511 | 1.0777 | 1.3177 | 0.4255 | 0.6005 | 0.6682 | 0.388291 tbnn_tu10 | 0.2128 | 0.1702 | 0.383 | 0.1418 | 0.2837 | 1.2766 | 0.2609 | 0.2857 | 0.735 | 0.1418 | 0.1773 | 0.3281 | 0.423141 tbnn_emb_tu10 | 1.0638 | 0.5957 | 0.4681 | 0.6525 | 0.9362 | 1.5035 | 1.0157 | 0.9195 | 1.1839 | 0.4681 | 0.5437 | 0.6237 | 0.400455 tbnn_dt2v_emb_tu10 | 0.6383 | 0.6809 | 0.4681 | 0.3688 | 1.0213 | 1.4468 | 0.6864 | 0.88 | 1.0638 | 0.2979 | 0.4662 | 0.5158 | 0.395372

uspt:

  | P_2 | P_5 | P_10 | r_2 | r_5 | r_10 | ndcg_2 | ndcg_5 | ndcg_10 | map_2 | map_5 | map_10 | aucroc -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- rrn | 0.0239 | 0.0383 | 0.0654 | 0.0140 | 0.0500 | 0.1370 | 0.0221 | 0.0408 | 0.0868 | 0.0096 | 0.0186 | 0.0340 | 0.5160 tbnn | 0.1843 | 0.1841 | 0.2029 | 0.0933 | 0.2321 | 0.5158 | 0.1794 | 0.2152 | 0.3481 | 0.0681 | 0.1056 | 0.1429 | 0.7544 tbnn_emb | 0.8272 | 0.7539 | 0.7042 | 0.3970 | 0.9021 | 1.6933 | 0.8457 | 0.9057 | 1.2657 | 0.3104 | 0.4533 | 0.5679 | 0.8359 tbnn_dt2v_emb | 1.2268 | 1.0583 | 0.9324 | 0.6037 | 1.2928 | 2.2518 | 1.2322 | 1.2960 | 1.7348 | 0.4626 | 0.6659 | 0.8118 | 0.8534 tbnn_tu1 | 0.0223 | 0.0239 | 0.0241 | 0.0119 | 0.0324 | 0.0668 | 0.0216 | 0.0286 | 0.0441 | 8.45E-03 | 0.0139 | 0.0184 | 0.496715 tbnn_emb_tu1 | 0.0622 | 0.0721 | 0.0777 | 0.0258 | 0.0804 | 0.168 | 0.0629 | 0.0804 | 0.1182 | 0.0189 | 0.033 | 0.0455 | 0.58621 tbnn_dt2v_emb_tu1 | 0.067 | 0.0756 | 0.0767 | 0.0239 | 0.07 | 0.1479 | 0.0695 | 0.0852 | 0.1136 | 0.02 | 0.0324 | 0.0439 | 0.603763 tbnn_tu5 | 0.1188 | 0.0788 | 0.0629 | 0.0561 | 0.0985 | 0.1634 | 0.1219 | 0.107 | 0.1332 | 0.0435 | 0.0553 | 0.0635 | 0.573823 tbnn_emb_tu5 | 0.1731 | 0.178 | 0.213 | 0.08 | 0.201 | 0.4709 | 0.1718 | 0.2038 | 0.3268 | 0.0588 | 0.0913 | 0.1324 | 0.676426 tbnn_dt2v_emb_tu5 | 0.1651 | 0.1822 | 0.1978 | 0.0616 | 0.1797 | 0.4014 | 0.162 | 0.1988 | 0.2881 | 0.0448 | 0.0782 | 0.1109 | 0.692164 tbnn_tu10 | 0.1077 | 0.0858 | 0.0739 | 0.0535 | 0.1052 | 0.1893 | 0.1075 | 0.108 | 0.1438 | 0.04 | 0.0543 | 0.065 | 0.630005 tbnn_emb_tu10 | 0.2824 | 0.2438 | 0.2549 | 0.1332 | 0.2824 | 0.5845 | 0.2824 | 0.2948 | 0.4301 | 0.1011 | 0.1424 | 0.1889 | 0.722787 tbnn_dt2v_emb_tu10 | 0.2361 | 0.246 | 0.2702 | 0.0992 | 0.2553 | 0.5662 | 0.23 | 0.2749 | 0.4075 | 0.0725 | 0.1166 | 0.1643 | 0.742351

gith:

  | P_2 | P_5 | P_10 | r_2 | r_5 | r_10 | ndcg_2 | ndcg_5 | ndcg_10 | map_2 | map_5 | map_10 | aucroc -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- rrn | 0.0000 | 0.0000 | 0.0990 | 0.0000 | 0.0000 | 0.3300 | 0.0000 | 0.0000 | 0.1466 | 0.0000 | 0.0000 | 0.0413 | 0.4503 tbnn | 1.7822 | 1.8218 | 1.8614 | 0.7795 | 1.6119 | 2.9388 | 1.8270 | 2.0523 | 2.6410 | 0.6259 | 0.9257 | 1.2584 | 0.5234 tbnn_emb | 1.4851 | 1.5446 | 1.4257 | 0.6930 | 1.5779 | 2.5944 | 1.5972 | 1.8324 | 2.2965 | 0.5785 | 0.8676 | 1.1009 | 0.5936 tbnn_dt2v_emb | 2.2772 | 2.2574 | 1.9010 | 0.9554 | 2.2220 | 3.4946 | 2.2996 | 2.6000 | 3.0724 | 0.7459 | 1.3180 | 1.5741 | 0.5632 tbnn_tu1 | 1.3861 | 1.3861 | 1.6436 | 0.3555 | 0.8322 | 1.9873 | 1.3861 | 1.5084 | 1.9793 | 0.3047 | 0.4508 | 0.6927 | 0.532643 tbnn_emb_tu1 | 1.6832 | 2.0198 | 1.8614 | 0.5275 | 1.2293 | 2.3821 | 1.7056 | 2.0098 | 2.3306 | 0.4022 | 0.6585 | 0.8941 | 0.583891 tbnn_dt2v_emb_tu1 | 0.8911 | 0.7921 | 1.0693 | 0.3719 | 0.7194 | 1.9331 | 0.8239 | 0.8746 | 1.491 | 0.2949 | 0.3968 | 0.6275 | 0.575736 tbnn_tu5 | 1.6832 | 1.5446 | 1.5446 | 0.5782 | 0.9709 | 1.74 | 1.7056 | 1.6671 | 1.9606 | 0.5171 | 0.6425 | 0.824 | 0.54547 tbnn_emb_tu5 | 1.8812 | 1.703 | 1.901 | 0.506 | 1.3586 | 2.4475 | 1.8588 | 1.9007 | 2.3953 | 0.3976 | 0.7551 | 0.9938 | 0.596085 tbnn_dt2v_emb_tu5 | 0.9901 | 1.3069 | 1.505 | 0.4403 | 1.134 | 2.4147 | 1.0797 | 1.4137 | 2.0665 | 0.4007 | 0.6537 | 0.9038 | 0.571246 tbnn_tu10 | 1.6832 | 1.5842 | 1.9208 | 0.376 | 0.9573 | 2.0696 | 1.6608 | 1.6583 | 2.1385 | 0.333 | 0.482 | 0.7566 | 0.555309 tbnn_emb_tu10 | 1.4851 | 1.7426 | 1.7426 | 0.59 | 1.4578 | 2.6825 | 1.4179 | 1.8435 | 2.3799 | 0.465 | 0.7389 | 0.967 | 0.603058 tbnn_dt2v_emb_tu10 | 1.2871 | 1.7822 | 1.901 | 0.3144 | 1.446 | 2.5824 | 1.2647 | 1.7632 | 2.3418 | 0.24 | 0.5707 | 0.8385 | 0.594359

[hosseinfani]

@VaghehDashti I think there is issues in the implementation. k=10 means that you select an expert from 10 years ago, which should be definitely a negative sample.

Also, when each year in the past does not help, not sure the union of all past years help.

Please create a toy example, put two disjoint set of experts in (a) 2010 and (b)2020 and 2021 years, then. train the model on 2020, picking negative samples from 2010. then predict for 2021. or something like this that shows the code is fine or reveal the bug in our logic of thinking.