pcgreat / SeqMatchSeq

Compare-Aggregate method for WikiQA (via PyTorch)
29 stars 7 forks source link

MAP value is really low compared to 0.727 (MAP) in the README #4

Closed anis016 closed 6 years ago

anis016 commented 6 years ago

Hello!

I tried to run the wikiqa here and I used the default settings as like it is present in the main.py However, my MAP and MRR is really low compared to what appears in the README. Could you please share how can I get the 0.727 (MAP)?

This is my output.

Building vocab dict ...
41985
Building wikiqa all data ...
Loading glove ...
Number of words not appear in glove: 5692
Vocal size: 41985
loading data ..
Loading data train...
Size of training data: 873
Loading data dev...
Size of dev data: 126
Loading embedding ...
100%|██████████| 88/88 [01:12<00:00,  1.21it/s]
Dev: MAP: 0.5656880523944017, MRR: 0.5696300612967281
100%|██████████| 88/88 [01:09<00:00,  1.27it/s]
Dev: MAP: 0.5698974342823551, MRR: 0.5726599899218948
100%|██████████| 88/88 [01:08<00:00,  1.28it/s]
Dev: MAP: 0.5589818727715553, MRR: 0.5617063492063491
100%|██████████| 88/88 [01:08<00:00,  1.28it/s]
Dev: MAP: 0.5333682080705892, MRR: 0.5356217525860384
100%|██████████| 88/88 [01:08<00:00,  1.28it/s]
Dev: MAP: 0.556504808389729, MRR: 0.5590680043060994
100%|██████████| 88/88 [01:08<00:00,  1.28it/s]
Dev: MAP: 0.596566708570677, MRR: 0.6045730283825524
100%|██████████| 88/88 [01:08<00:00,  1.28it/s]
Dev: MAP: 0.5831342378961426, MRR: 0.5913074359502931
100%|██████████| 88/88 [01:08<00:00,  1.28it/s]
  0%|          | 0/88 [00:00<?, ?it/s]Dev: MAP: 0.5700577508910841, MRR: 0.5774248458772269
100%|██████████| 88/88 [01:08<00:00,  1.28it/s]
Dev: MAP: 0.5925723121547828, MRR: 0.6021286033190795
100%|██████████| 88/88 [01:09<00:00,  1.27it/s]
Dev: MAP: 0.5701876569932125, MRR: 0.5777376944043611
anis016 commented 6 years ago

never mind! I somehow overlooked this discussion. https://github.com/pcgreat/SeqMatchSeq/issues/1