search

wjhou / ORGan

Code for the paper "ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning" (ACL'23).
https://wjhou.github.io/ORGan/
Apache License 2.0
48 stars 0 forks source link

CE scores #4

Closed Yingshu-Li closed 6 months ago

Yingshu-Li commented 10 months ago

Hi,

are the scores reported in this paper macro, micro or macro-based F1 scores?

Thanks a lot in advance!

wjhou commented 10 months ago

Hi,

We use macro-weighted metrics for evaluation. You may also read the comment from this issue https://github.com/zhjohnchan/R2GenCMN/issues/12.

Best, Ethan

Yingshu-Li commented 7 months ago

Hi,

I have tried your mentioned method. I used chexpert to label the report. I cannot reproduce the result. The scores are much lower.

here is my method:

  1. use chexbert to label the report. https://github.com/stanfordmlgroup/CheXbert
  2. set label: nan and -1 to 0.
  3. use this script to calculate the score. https://github.com/zhjohnchan/R2GenCMN/blob/main/compute_ce.py

Could you please help me to check if I have made any mistakes?

best

wjhou commented 7 months ago

Hi,

We set -1 (i.e., Uncertain) to 1 in the calculation, similar to https://github.com/zzxslp/WCL/blob/main/chexpert-labeler/calculate_metric.py.

Could you provide the scores (NLG and CE scores) you obtained?

Best, Ethan