search

behavioral-data / Empathy-Mental-Health

Repository containing codes and dataset access instructions for the EMNLP 2020 paper on empathy in text-based mental health support
154 stars 34 forks source link

Reproducing rationale extraction #4

Closed bckim92 closed 3 years ago

bckim92 commented 3 years ago

Hi, thank you for your amazing work.

I might miss something, but I have some trouble with reproducing rationale extraction results. Can you give me more training details? Or can I get pre-trained weights? The followings are my rerun with default hyperparameters. I randomly split the dataset into train and dev sets (9:1).

(Emotional Reactions)

Running validation...

  Accuracy-Empathy: 0.7856
  macro_f1_empathy: 0.7638
  Accuracy-Rationale: 0.8923
  IOU-F1-Rationale: 0.0542
  macro_f1_rationale: 0.0595

(Interpretations)

Running validation...

  Accuracy-Empathy: 0.8494
  macro_f1_empathy: 0.6573
  Accuracy-Rationale: 0.8788
  IOU-F1-Rationale: 0.0952
  macro_f1_rationale: 0.1353

(Explorations)

Running validation...

  Accuracy-Empathy: 0.9106
  macro_f1_empathy: 0.7172
  Accuracy-Rationale: 0.9427
  IOU-F1-Rationale: 0.0291
  macro_f1_rationale: 0.0485

I really appreciate any help you can provide.

ash-shar commented 3 years ago

Hi,

Thanks so much for highlighting this!

It turns out that there was a small bug in the data processing script (dataset/process_data.py) which caused this issue. This should now be fixed. You will have to process your input files again though. Let me know if you still can't reproduce the numbers.

Thanks again!

bckim92 commented 3 years ago

Hi, thank you for the response! Now it works completely well. Thanks again, and have a good day 👍