search

WenzhengZhang / EntQA

Pytorch implementation of EntQA paper
MIT License
60 stars 12 forks source link

some issues to reproduce the Gerbil result #3

Closed yifding closed 2 years ago

yifding commented 2 years ago

Hello, really nice work:

I have a few questions/corrections/suggestions to reproduce the gerbil results.

  1. the "pip -r install requirements.txt" in readme should be "pip install -r requirements.txt"
  2. 'torch=1.7.1' in requirements.txt shoud be 'torch==1.7.1'
  3. in readme, the parameter '--raw_kb_dir' command should be '--raw_kb_path' used by preprocess_data.py
  4. As for gerbil evaluation, the EntQA package uses similar interface to "end2end_neural_el" and "rel". I am trying to use the provided gerbil snapshot to evaluate on "rel" locally, but find the metrics is much lower than its reported values. It shows many wikipedia pages are unknown which should be valid pages. Do you have any ideas ?
  5. Can you explain the actual input/output (an example is greatly appreciated) of the "python gerbil_experiments/server.py" if it is setup.
WenzhengZhang commented 2 years ago

Thanks for pointing out the mistakes in readme for me! I've never tried "rel" with Gerbil, so I have no idea why you met those problems with the provided gerbil snapshot. From my experience on EntQA, Gerbil usually reports slightly higher performance than running local evaluation on AIDA test. While you're running "gerbil_experiments/server.py", Gerbil will feed a raw text passage to my model each time, my model will output a list of (start_idx, end_idx, entity_name) for that raw text passage to Gerbil. After the experiment is finished, you can find the experiments results on "http://localhost:1234/gerbil".

yifding commented 2 years ago

Thank you!