search

imranraad07 / BugReportQA

0 stars 0 forks source link

improve utility calculation #50

Closed damevski closed 4 years ago

damevski commented 4 years ago

Things to look into:

  1. My planned formulation was as follows (the implementation does not follow this): U(p+a) = softmax(d_OB+d_EB+dS2R), where dOB = OBa_i - OBp, and similarly dEB and dS2R OB_x is the number of OB sentences in document x, as defined by Chaparro et al.
  2. Currently the utility calculation is not part of the learning process. Is this what we want
  3. We could make the evaluation more limited e.g., evaluate only whether the correct answer is chosen given the question and the post, i.e., p(a|p,q). Argue utility is separate and will be evaluated only by a user study.
damevski commented 4 years ago

Assuming we do #54, we can ignore most of this issue.