MAP and MRR equals 0

[jackyuanjie1990]

Hi galsang, thank you for sharing code. I run your codes following the README. The cost tends to reduce like this: [Epoch 4] ('[batch 100] cost:', 0.40089718) ('[batch 200] cost:', 0.27272141) ('[batch 300] cost:', 0.16169518) ('model saved as', './models/WikiQA-BCNN-2-4') ('LR saved as', './models/WikiQA-BCNN-2-4-LR.pkl') ('SVM saved as', './models/WikiQA-BCNN-2-4-SVM.pkl')

However, when I test, MAP and MRR equal 0. Moreover, the result file is empty. Do you know how to solve this problems or what need I to focus when I run the codes. Thank you very much！

('[Epoch 1] MAP:', 0, '/ MRR:', 0) ('./models/WikiQA-BCNN-2-2', 'restored.') ('./models/WikiQA-BCNN-2-2-LR.pkl', 'restored.') ('[Epoch 2] MAP:', 0, '/ MRR:', 0) ('./models/WikiQA-BCNN-2-3', 'restored.') ('./models/WikiQA-BCNN-2-3-LR.pkl', 'restored.') ('[Epoch 3] MAP:', 0, '/ MRR:', 0) ('./models/WikiQA-BCNN-2-4', 'restored.') ('./models/WikiQA-BCNN-2-4-LR.pkl', 'restored.')

[galsang]

Hi,

How about retrying running the code after cleaning /models and /experiments folders? There are some files made from my own experiments.

Actually, after training is done, models are saved automatically in /models and after testing is done results are saved in /experiments.

I'll also try running the code in a different environment where I haven't developed the code. (Please tell me which environment(Linux, windows, else..) you are using.) Thank you.

[jackyuanjie1990]

Actually,before running your code, I have cleaned models and experiments folders. I use ubuntu system, python 2.7, tensorflow 1.0.1. I just change the code "with open("./WikiQA_QA/Wiki-"+mode+".txt","r",encoding ="urged-8") as f" to "with open("./WikiQA_QA/Wiki-"+mode+".txt","r") " because python 2.7 doesn't support the former. Do you think the version of python will affect result? Thank you very much!

[jackyuanjie1990]

Hi, Kim. I have another question. Did you consider the question about data imbalance? Because the number of unrelated QA pairs is much larger than related QA pairs. Thank you very much!

[galsang]

Dear @jackyuanjie1990, I want to tell you that I've verified the fact that my code runs well in other environments(I tested on Ubuntu 14.04.1, python3, only with CPU support.) I highly recommend you to run the code on Python3 because there may be latent errors when you run the code written in Python 3 on Python 2 environments.

And I didn't consider the thing you mentioned(data imbalance problem) because that's the WikiQA dataset itself. I didn't modify the dataset because I want to reproduce results as the authors suggested.

Thank you.

[jackyuanjie1990]

Thank you for your patient reply! I will try Python3.

[jackyuanjie1990]

Hi Kim. I get the same result as you post after I change python 2.7 to 3.5. You are right, it really has latent errors when run this code under python 2.7. Anyway, thank you very much again.

[wangzil]

The reason is that MAP and MRR is initialized to int 0(python 2.7)，so change it to float(0),and you can get the correct result.

[galsang]

@wangzil thanks for your correction.