Data from the paper ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI by Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser.
Please cite as:
@inproceedings{cercas-curry-etal-2021-convabuse,
title = "{C}onv{A}buse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational {AI}",
author = "Cercas Curry, Amanda and
Abercrombie, Gavin and
Rieser, Verena",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.587",
doi = "10.18653/v1/2021.emnlp-main.587",
pages = "7388--7403"
}
We provide two versions of the dataset:
All annotations are presented in binary format (1/0). Annotators only labelled secondary tasks (e.g. abuse type, target etc.) in cases they had considered to be abusive (i.e. -1/-2/-3).
The columns are:
Column | Column header | Explanation |
---|---|---|
0. | example_no |
|
1. | annotator_id |
Annotator ID |
2. | conv_id |
Conversation ID |
3. | prev_agent |
Agent's previous utterance |
4. | prev_user |
User's previous utterance |
5. | agent |
Agent utterance |
6. | user |
User (target) utterance |
7. | bot |
Agent name (CarbonBot/Eliza) |
8. | is_abuse.1 |
Not abusive |
9. | is_abuse.0 |
Ambiguous |
10. | is_abuse.-1 |
Mildly abusive |
11. | is_abuse.-2 |
Strongly abusive |
12. | is_abuse.-3 |
Very strongly abusive |
13. | type.ableism |
Type: Ableism |
14. | type.homophobic |
Type: Homophobic |
15. | type.intellectual |
Type: Intellectual |
16. | type.racist |
Type: Racist |
17. | type.sexism |
Type: Sexist |
18. | type.sex_harassment |
Type: Sexual harassment |
19. | type.transphobic |
Type: Transphobic |
20. | target.generalised |
Target: General |
21. | target.individual |
Target: Individual |
22. | target.system |
Target:system/agent |
23. | directness.explicit |
Directness: Explicit |
24. | directness.implicit |
Directness: Implicit |
In (2), the dataset is divided into train, vailidation, and test splits.
(2) contains the same fields as (1), but each row includes all the annotators responses, with the annotator response column headers preceded by the annotator ID, e.g. Annotator1.is_abuse.-1
.
Column | Column header |
---|---|
0. | example_id |
1. | conv_id |
2. | prev_agent |
3. | prev_user |
4. | agent |
5. | user |
6. | bot |
7. | Annotator1_is_abuse.1 |
8. | Annotator1_is_abuse.0 |
9. | Annotator1_is_abuse.-1 |
10. | Annotator1_is_abuse.-2 |
11. | Annotator1_is_abuse.-3 |
12. | Annotator1_type.ableist |
13. | Annotator1_type.homophobic |
14. | Annotator1_intellectual |
15. | Annotator1_racist |
16. | Annotator1_sexism |
17. | Annotator1_sex_harassment |
18. | Annotator1_transphobic |
19. | Annotator1_target.generalised |
20. | Annotator1_target.individual |
21. | Annotator1_target.system |
22. | Annotator1_explicit |
23. | Annotator1_implicit |
24. | Annotator2_is_abuse.1 |
... | ... |
142. | Annotator8_implicit |
Note that for privacy reasons, we provide data from CarbonBot and E.L.I.Z.A only. We are unable to release the examples from Alana used in the paper.