Open 2399240664 opened 1 year ago
What do you mean :)
Now, I can only get the experimental results of all types of questions (Overall) by running Exact. I cannot obtain experimental data of other types of questions, such as Explicit, Implicit, Temp Ans., Ordinal.
I want to know how I need to operate to obtain these experimental data.
You can use the type annotation in the benchmark to compute P@1, etc, for each category from the overall results. For example, if you know a question falling into explicit type, and its P@1, etc, you can compute the average P@1 for all questions with explicit type.
Sorry, In the Exaqt code, I did not find the type annotation are used to calculate P@1.
I will provide the code soon :)
The evaluate.py is updated. Now the performance for each category can be computed. Please let me know if there are any problems.
Thank you.
How to run Explicit, Implicit, Temp Ans., Ordinal's experimental data?