Detect and prevent toxicity in models' output

[marco-c]

We might be able to find open source toxicity lists.

[marco-c]

Things we discussed:

• Add filtering on the training datasets to ignore sentence pairs which have bad words in target but no bad words in source
  • Maybe do something at inference too?
  • If a toxic translation is generated, generate N sentences and see if one doesn't have toxic content.
• Toxicity statistics in tasks.
  • See how much toxic content is in parallel content, source, target, output statistics and sampling.

[gregtatum]

Toxicity-200 https://github.com/facebookresearch/flores/blob/main/toxicity/README.md