search

hotpotqa / hotpot

Apache License 2.0
445 stars 75 forks source link

Using own retrieval results #33

Open AndresFRJ98 opened 4 years ago

AndresFRJ98 commented 4 years ago

Hello. I am attempting to experiments with using my own retrieval results and assessing the impact on the performance of the model (for the full-wiki setting)

If I understood correctly, the process would be as follows.

Generate my own version of hotpot_dev_fullwiki_v1.json in which the context entry of the JSON has been modified with my own paragraphs. All the other keys should be left identical. I shall refer to this json file as custom_dev_fullwiki.json.

Likewise for the test set, to create custom_test_fullwiki.json

Then, I would run the preprocessing , training and evaluation commands as specified in the README.

To make meaningful comparison of metrics, once I change my retrieval method and generate a 'v2.json' set of files, I should start again from scratch on preprocessing, then train etc.

Is this correct?

ManuelRios18 commented 4 weeks ago

Yeah, I think you are right :)