rookie-joe / AutoPSV

30 stars 0 forks source link

A little bit confused about the paper #2

Open waltonfuture opened 2 weeks ago

waltonfuture commented 2 weeks ago

Thanks for your interesting work. I have a few questions.

  1. In the right part of Fig 1, why is +0.25 for step2? It seems that it should be $\frac{0.35-0.27}{0.35} = 0.29$?
  2. After using first error location to get the process labels, we can train PSV model. In the evaluation in Section 5.1, do we need to calculate $\Delta_{conf}^{t}$ using PSV model for each step or just use the output score from PSV model for each step?

Looking forward to your responses.

rookie-joe commented 1 week ago

Thank you for your careful review of our work, and I apologize for the delayed response.

  1. Regarding Fig 1 and the +0.25 value in step 2: You are correct - this is actually an error in our diagram. We plan to revise this in the next version of the paper.

  2. About the process labeling and PSV model evaluation: As described in Section 3.3 of our paper, we use the "first error location" strategy. To clarify the process:

    • Once you find the first error location using the threshold θ:
      • If Δconf_t > θ: yi_t = 1
      • Otherwise: yi_t = 0 and for all subsequent steps t', yi_t' = 0

    Therefore, after identifying the first error location, you don't need to calculate the delta confidence scores for the following steps.

Please let me know if you have any other questions!