search

ahans30 / Binoculars

[ICML 2024] Binoculars: Zero-Shot Detection of LLM-Generated Text
https://arxiv.org/abs/2401.12070
BSD 3-Clause "New" or "Revised" License
189 stars 26 forks source link

How to compute AUROC using the Binocular scores? #4

Closed AmritaBh closed 6 months ago

AmritaBh commented 6 months ago

The additional results in the appendix section of the paper report AUC scores. How do you compute the AUROC scores using the binocular scores? A script to do this or pseudocode would be super helpful. Thanks!

ahans30 commented 6 months ago

Hi, thanks for your interest in the project. Below is code snippet you can use to compute AUC using standard pandas and sklearn libs.

from sklearn import metrics

# df is pandas dataframe with ground truth `sample_class` (1 for AI-generated, 0 for human-generated text)
score = "binoculars_score"
label = "sample_class"

df[score] *= -1 # reverse scale so that higher score indicates positive/AI-generated class
fpr, tpr, _ = metrics.roc_curve(y_true=df[label], y_score=df[score], pos_label=1)
roc_auc = metrics.auc(fpr, tpr)

Hopefully, this should help you out. I'm closing this issue, but please feel free to comment if you further face any issues.