Shilin-LU
commented
1 week ago Hi, thank you for your interest!
-
TPR@0.1%FPR is a statistical metric that applies to a set of images rather than individual images. Therefore, it isn’t included in our single-image demo. However, we plan to release this metric alongside our full benchmark once our work is accepted, so please stay tuned!
-
Decoded accuracy does not need to be 100%. To put it simply: If a model decodes unwatermarked images with about 50% accuracy and watermarked images with 75% accuracy, statistical testing can likely classify the images correctly. However, if a model frequently achieves high bit accuracy on unwatermarked images, even high accuracy on watermarked images could signal a high false positive rate, indicating poor performance as a watermarking model.
-
Why some methods achieve high bit accuracy but low TPR@0.1%FPR: This occurs when methods often misclassify unwatermarked images as watermarked, mistakenly decoding a message from unwatermarked images.
-
ROC may be of interest to you for understanding TPR@0.1%FPR. For a simple implementation: calculate the decoding accuracies of both unwatermarked and watermarked images, plot the ROC, and use the following code to determine TPR at various FPRs (Alternatively, you can wait for our upcoming benchmark, which will include complete metric calculation codes):
from sklearn import metrics
def compute_auroc_tpr_fpr(unwmed, wmed):
fpr, tpr, thresholds = metrics.roc_curve(unwmed, wmed, pos_label=1)
auc = metrics.auc(fpr, tpr)
tpr_at_1_fpr = tpr[np.where(fpr < 0.01)[0][-1]]
tpr_at_01_fpr= tpr[np.where(fpr < 0.001)[0][-1]]
return auc, tpr_at_1_fpr, tpr_at_01_fpr
WWEISONG
Hi Shilin,
It is a great work, and thanks for releasing it to the public.
I was confused about the TPR@0.1%FPR, and from your code, it seems there is only bit accuracy. Could you please indicate how you calculate TPR? Does it require the decoded watermark exactly match the secret (say 100% bit accuracy)?