TF-PII Redaction Benchmark

[loubnabnl]

Build a benchmark (for short term) for PII detection of Emails, IP addresses and SSH & API keys:

• [ ] Bibliographic research for existing PII benchmarks
• [ ] Setup the benchmark to be annotated 100~1000 samples from the Stack (from the most popular languages)
• [ ] Setup an Annotation platform (https://www.lighttag.io/pricing/ vs https://labelstud.io)

[loubnabnl]

Found this paper for AWS credentials detection in Java files, we can test something similar, they report 100% recall and 91% precision but the test set is not large and from 2015 (it's not publicly available either)

• They first restrict their search set to files that contain the literal BasicAWSCredentials.
• Then they use two regexes to find Client ID and Secret keys.
• They observed false positives for keys and added a heuristic-driven filtering:
  • Look for instances where a match for the Client ID and a match for the Secret Key appear within 5 lines of each other
  • Use a password-strength estimator to remove false positives that are human-readable strings

[loubnabnl]

Here is the benchmark of 400 samples we annotated for: Emails, IP addresses, SSH & API keys, Names and Usernames.