실험결과

[maygodwithu]

30 epoch

snrm

[Iter-65490 Loss-0.003]: Validation: normalized_discounted_cumulative_gain@3(0.0): 0.4048 - normalized_discounted_cumulative_gain@5(0.0): 0.4867 - mean_average_precision(0.0): 0.4441 Epoch 30/30: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 2183/2183 [05:31<00:00, 6.59it/s, loss=0.001] Cost time: 9796.057008266449s

matchpyramid

[Iter-104790 Loss-0.001]: Validation: normalized_discounted_cumulative_gain@3(0.0): 0.4872 - normalized_discounted_cumulative_gain@5(0.0): 0.5571 - mean_average_precision(0.0): 0.5135 Epoch 30/30: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 3493/3493 [05:15<00:00, 11.07it/s, loss=0.000] Cost time: 9339.508510112762s

10 epoch wiki-qa

Validation: normalized_discounted_cumulative_gain@3(0.0): 0.5383 - normalized_discounted_cumulative_gain@5(0.0): 0.6051 - mean_average_precision(0.0): 0.5477

10 epoch TREC

Validation: normalized_discounted_cumulative_gain@3(0.0): 0.4624 - normalized_discounted_cumulative_gain@5(0.0): 0.5351 - mean_average_precision(0.0): 0.4952

[maygodwithu]

Full MSMARCO data 10 epoch

Matchpyramid

(qc cut = 5) Validation: normalized_discounted_cumulative_gain@3(0.0): 0.5959 - normalized_discounted_cumulative_gain@5(0.0): 0.6566 - mean_average_precision(0.0): 0.6013 Epoch 10/10: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 34921/34921 [44:11<00:00, 13.17it/s, loss=0.000] Cost time: 34729.66079545021s

( qc cut = 3) Validation: normalized_discounted_cumulative_gain@3(0.0): 0.5243 - normalized_discounted_cumulative_gain@5(0.0): 0.593 - mean_average_precision(0.0): 0.5407 Epoch 5/5: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34922/34922 [1:03:36<00:00, 9.15it/s, loss=0.000] Cost time: 28418.586758613586s

ARCi

Validation: normalized_discounted_cumulative_gain@3(0.0): 0.5002 - normalized_discounted_cumulative_gain@5(0.0): 0.5682 - mean_average_precision(0.0): 0.5245 Epoch 10/10: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 34921/34921 [43:04<00:00, 13.51it/s, loss=0.412] Cost time: 28308.64152264595s Validation: normalized_discounted_cumulative_gain@3(0.0): 0.4474 -

SNRM

normalized_discounted_cumulative_gain@5(0.0): 0.5238 - mean_average_precision(0.0): 0.4812 Epoch 10/10: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 21826/21826 [1:03:19<00:00, 5.74it/s, loss=0.199] Cost time: 35095.056727170944s

[maygodwithu]

kurtois

naive kurtois

truth = 18.38959458961045 not truth = 11.714694422016255

kurtois * positive ratio

truth = 12.247806981656643 not truth = 6.932621278367608

kurtois ( 10%)

truth = 1.0562272664417323 not truth = 0.7010201578159544

[maygodwithu]

Stat - correlation

score_mean= 2.492006588409223 score_std= 7.66333713953733 matching_mean= 0.25249153 matching_std= 0.25249153 cam_mean= 0.0021031518 cam_std= 0.014975903 corr_mean= 0.8634431857261743 corr_std= 0.15398663321834727 pearsoncorr_mean= 0.1967436108350385 pearsoncorr_std= 0.0669515239695644

stat - new ( 10% )

score_mean= 2.9577996340935906 score_std= 8.511598094908088 interaction_mean= 0.11165669 interaction_std= 0.11165669 cam_mean= nan cam_std= nan corr_mean= 0.821430919701612 corr_std= 0.25182115119973003 spearman_corr_mean= 0.2319838978488567 spearman_corr_std= 0.07926738810864235

[maygodwithu]

negative word : 9537

oil for dry hair 이므로, jojoba, aromatherapy, fragrance는 관계가 적다. (neural_env) jkchoi@rtx2080ti4:~/neuralranking/MatchZoo-py/tutorials/ranking/eval$ python3 margin_dterm.py 9537 vocab read term of 0 index = 9537 sandalwood helps experts ['what', 'kind', 'of', 'oil', 'is', 'good', 'for', 'dry', 'hair'] jojoba -398.0 aromatherapy -373.0 fragrance -372.0 instructions -340.0 industry -334.0 argan -324.0 oily -276.0 slideshowslideshow -253.0 over -250.0 may -241.0 tell -233.0 do -219.0 only -217.0 improves -216.0 myth -216.0 what -213.0 products -211.0 other -211.0 eventually -207.0 skin -189.0

negative word : 50

-->모두 관계되어 있으므로, prevention 만 negative word 라고 .. (neural_env) jkchoi@rtx2080ti4:~/neuralranking/MatchZoo-py/tutorials/ranking/eval$ python3 margin_dterm.py 50 vocab read term of 0 index = 50 infections infections a borne infection may ['blood', 'diseases', 'that', 'are', 'sexually', 'transmitted'] prevention -92.0 herpes -78.0 gonorrhea -70.0 contaminated -49.0 example -46.0 syphilis -45.0 health -40.0 other -40.0 from -39.0 semen -33.0 and -29.0

[maygodwithu]

snippet 개선 예제

num= 7084 naive pos= (26, 46, 2) cam pos= (426, 446, 2, 2.027264118245512) query= ['what', 'is', 'mean', 'streaming'] naive snippet= ['the', 'act', 'the', 'process', 'or', 'an', 'instance', 'of', 'streaming', 'data', 'see', '2', '3', 'or', 'of', 'accessing', 'data', 'that', 'is', 'being'] cam snippet= ['english', 'language', 'definition', 'of', 'streaming', 'for', 'english', 'language', 'playing', 'continuously', 'as', 'data', 'is', 'sent', 'to', 'a', 'computer', 'over', 'the', 'internet']