TwiBot-22
This is the official repository of TwiBot-22 @ NeurIPS 2022, Datasets and Benchmarks Track. This dataset is collected from the Twitter website before 2022.
Introduction
TwiBot-22 is the largest and most comprehensive Twitter bot detection benchmark to date. Specifically, TwiBot-22 is designed to address the challenges of limited dataset scale, imcomplete graph structure, and low annotation quality in previous datasets. For more details, please refer to the TwiBot-22 paper and statistics.
Dataset Format
Each dataset contains node.json (or tweet.json, user.json, list.json, and hashtag.json for TwiBot-22), label.csv, split.csv and edge.csv (for datasets with graph structure). See here for a detailed description of these files.
How to download TwiBot-22 dataset
TwiBot-22 is available at Google Drive.
Please apply for access by contacting shangbin at cs.washington.edu with your institutional email address and clearly state your institution, your research advisor (if any), and your use case of TwiBot-22.
How to download other datasets
For TwiBot-20, visit the TwiBot-20 github repository.
For other datasets, please visit the Bot Repository.
After downloading these datasets, you can transform them into the 4-file format detailed in "Dataset Format". Alternatively, you can directly download our preprocessed version:
For TwiBot-20, visit the TwiBot-20 github repository, apply for TwiBot-20 access, and there will be a TwiBot-20-Format22.zip in the TwiBot-20 Google Drive link.
For other datasets, you can directly download them from Google Drive. You should adhere to the license of each dataset, the "Content redistribution" section of the Twitter Developer Agreement and Policy, the rules set by the Bot Repository, and only use these datasets for research purposes.
Requirements
- pip:
pip install -r requirements.txt - conda :
conda install --yes --file requirements.txt
How to run baselines
- clone this repo by running
git clone https://github.com/LuoUndergradXJTU/TwiBot-22.git - make dataset directory
mkdir datasetsand download datasets to./datasets - change directory to
src/{name_of_the_baseline} - run experiments under the guidance of corresponding
readme.md
Baseline Overview
| baseline | paper | acc on Twibot-22 | f1 on Twibot-22 | type | tags |
|---|---|---|---|---|---|
| Abreu et al. | link | 0.7066 | 0.5344 | F | random forest |
| Alhosseini et al. | link | 0.4772 | 0.3810 | F G | gcn |
| BGSRD | link | 0.7188 | 0.2114 | F | BERT GAT |
| Bot Hunter | link | 0.7279 | 0.2346 | F | random forest |
| Botometer | link | 0.4987 | 0.4257 | F T G | |
| BotRGCN | link | 0.7966 | 0.5750 | F T G | BotRGCN |
| Cresci et al. | link | - | - | T | DNA |
| Dehghan et al. | link | - | - | F T G | Graph |
| Efthimion et al. | link | 0.7408 | 0.2758 | F T | efthimion |
| EvolveBot | link | 0.7109 | 0.1409 | F T G | random forest |
| FriendBot | link | - | - | F T G | random forest |
| Kipf et al. | link | 0.7839 | 0.5496 | F T G | Graph Neural Network |
| Velickovic et al. | link | 0.7948 | 0.5586 | F T G | Graph Neural Network |
| GraphHist | link | - | - | F T G | random forest |
| Hayawi et al. | link | 0.7650 | 0.2474 | F | lstm |
| HGT | link | 0.7491 | 0.3960 | F T G | Graph Neural Networks |
| SimpleHGN | link | 0.7672 | 0.4544 | F T G | Graph Neural Networks |
| Kantepe et al. | link | 0.7640 | 0.5870 | F T | random forest |
| Knauth et al. | link | 0.7125 | 0.3709 | F T G | random forest |
| Kouvela et al. | link | 0.7644 | 0.3003 | F T | random forest |
| Kudugunta et al. | link | 0.6587 | 0.5167 | F | SMOTENN, random forest |
| Lee et al. | link | 0.7628 | 0.3041 | F T | random forest |
| LOBO | link | 0.7570 | 0.3857 | F T | random forest |
| Miller et al. | link | 0.3037 | 0.4529 | F T | k means |
| Moghaddam et al. | link | 0.7378 | 0.3207 | F G | random forest |
| NameBot | link | 0.7061 | 0.0050 | F | Logistic Regression |
| RGT | link | 0.7647 | 0.4294 | F T G | Graph Neural Networks |
| RoBERTa | link | 0.7207 | 0.2053 | F T | RoBERTa |
| Rodriguez-Ruiz | link | 0.4936 | 0.5657 | F T G | SVM |
| Santos et al. | link | - | - | F T | decision tree |
| SATAR | link | - | - | F T G | |
| SGBot | link | 0.7508 | 0.3659 | F T | random forest |
| T5 | link | 0.7205 | 0.2027 | T | T5 |
| Varol et al. | link | 0.7392 | 0.2754 | F T | random forest |
| Wei et al. | link | 0.7020 | 0.5360 | T |
where - represents the baseline could not scale to TwiBot-22 dataset
Precision
| Precision | Botometer-feedback-2019 | Cresci-2015 | Cresci-2017 | Cresci-rtbust-2019 | Cresci-stock-2018 | Gilani-2017 | Midterm-2018 | Twibot-20 | Twibot-22 |
|---|---|---|---|---|---|---|---|---|---|
| Abreu et al. | 63.63 $_{3.60}$ | 99.05 $_{0.21}$ | 98.34 $_{0.13}$ | 78.57 $_{1.44}$ | 75.45 $_{0.45}$ | 76.82 $_{1.20}$ | 97.28 $_{0.07}$ | 72.20 $_{0.52}$ | 50.92 $_{0.10}$ |
| Alhosseini et al. | - | 87.69 $_{1.23}$ | - | - | - | - | - | 57.81 $_{0.43}$ | 29.99 $_{3.08}$ |
| BGSRD | 27.50 $_{28.2}$ | 86.52 $_{0.64}$ | 75.85 $_{0.00}$ | 58.13 $_{11.1}$ | 52.78 $_{0.75}$ | 25.43 $_{23.2}$ | 84.40 $_{0.93}$ | 67.64 $_{2.26}$ | 22.55 $_{30.9}$ |
| BotHunter | - | 98.55 $_{0.56}$ | 98.65 $_{0.05}$ | 81.92 $_{2.04}$ | 84.29 $_{0.10}$ | 78.99 $_{0.96}$ | 99.44 $_{0.15}$ | 72.77 $_{0.25}$ | 68.09 $_{0.36}$ |
| Botometer | 21.05 - | 50.54 - | 93.35 - | 65.22 - | 68.50 - | 62.99 - | 31.18 - | 55.67 - | 30.81 - |
| BotRGCN | - | 95.51 $_{1.02}$ | - | - | - | - | - | 84.52 $_{0.54}$ | 74.81 $_{2.22}$ |
| Cresci | - | 0.59 - | 12.96 - | - | - | - | - | 7.66 - | - |
| Dehghan et al. | - | 96.15 $_{0.00}$ | - | - | - | - | - | 94.72 $_{0.00}$ | - |
| Efthimion et al. | 0.00 $_{0.00}$ | 93.82 $_{0.00}$ | 94.58 $_{0.00}$ | 68.29 $_{0.00}$ | 82.75 $_{0.00}$ | 37.50 $_{0.00}$ | 98.01 $_{0.00}$ | 64.20 $_{0.00}$ | 77.78 $_{0.00}$ |
| EvolveBot | - | 85.03 $_{3.77}$ | - | - | - | - | - | 66.93 $_{0.60}$ | 56.38 $_{0.40}$ |
| FriendBot | - | 95.29 $_{1.62}$ | 77.55 $_{0.81}$ | - | - | - | - | 72.64 $_{0.52}$ | - |
| GCN | - | 95.59 $_{0.69}$ | - | - | - | - | - | 75.23 $_{3.08}$ | 71.19 $_{1.28}$ |
| GAT | - | 96.10 $_{0.71}$ | - | - | - | - | - | 81.39 $_{1.18}$ | 76.23 $_{1.39}$ |
| GraphHist | - | 73.12 $_{0.10}$ | - | - | - | - | - | 51.27 $_{0.20}$ | - |
| Hayawi et al. | 25.00 $_{0.06}$ | 92.96 $_{0.03}$ | 95.47 $_{0.01}$ | 48.82 $_{0.01}$ | 50.73 $_{0.03}$ | 51.44 $_{0.05}$ | 85.30 $_{0.00}$ | 71.61 $_{0.01}$ | 80.00 $_{0.27}$ |
| HGT | - | 94.80 $_{0.49}$ | - | - | - | - | - | 85.55 $_{0.31}$ | 68.22 $_{2.71}$ |
| SimpleHGN | - | 95.68 $_{0.90}$ | - | - | - | - | - | 84.76 $_{0.46}$ | 72.57 $_{2.79}$ |
| Kantepe et al. | - | 81.30 $_{1.40}$ | 83.00 $_{0.90}$ | - | - | - | - | 63.40 $_{2.10}$ | 78.60 $_{1.80}$ |
| Knauth et al. | 57.41 $_{0.00}$ | 85.70 $_{0.00}$ | 91.56 $_{0.00}$ | 57.41 $_{0.00}$ | 99.89 $_{0.00}$ | 35.17 $_{0.00}$ | 99.91 $_{0.00}$ | 96.56 $_{0.00}$ | - |
| Kouvela et al. | 48.00 $_{4.47}$ | 99.54 $_{0.18}$ | 99.24 $_{0.13}$ | 82.27 $_{2.00}$ | 82.17 $_{0.46}$ | 79.69 $_{1.09}$ | 97.56 $_{0.04}$ | 79.33 $_{0.44}$ | 69.30 $_{0.14}$ |
| Kudugunta et al. | 56.67 $_{10.8}$ | 100.0 $_{0.00}$ | 98.53 $_{0.19}$ | 66.09 $_{2.35}$ | 54.87 $_{0.47}$ | 85.44 $_{2.42}$ | 99.06 $_{0.16}$ | 80.40 $_{0.60}$ | 44.31 $_{0.00}$ |
| Lee et al. | 58.97 $_{3.29}$ | 98.65 $_{0.14}$ | 99.56 $_{0.08}$ | 79.37 $_{2.97}$ | 84.75 $_{0.42}$ | 77.58 $_{1.31}$ | 97.36 $_{0.07}$ | 76.60 $_{0.37}$ | 67.23 $_{0.29}$ |
| LOBO | - | 98.47 $_{0.63}$ | 99.30 $_{0.08}$ | - | - | - | - | 74.83 $_{0.08}$ | 75.43 $_{0.15}$ |
| Miller et al. | 0.00 $_{0.00}$ | 72.07 $_{0.00}$ | 77.21 $_{0.18}$ | 52.17 $_{0.00}$ | 54.78 $_{0.00}$ | 48.89 $_{0.00}$ | 83.85 $_{0.00}$ | 60.71 $_{0.20}$ | 29.46 $_{0.00}$ |
| Moghaddam et al. | - | 98.33 $_{0.26}$ | - | - | - | - | - | 72.29 $_{0.67}$ | 67.61 $_{0.10}$ |
| NameBot | 45.45 $_{0.00}$ | 76.81 $_{0.00}$ | 80.39 $_{0.03}$ | 65.00 $_{0.00}$ | 58.34 $_{0.00}$ | 58.21 $_{0.00}$ | 86.93 $_{0.00}$ | 58.72 $_{0.00}$ | 67.73 $_{0.00}$ |
| RGT | - | 96.38 $_{0.59}$ | - | - | - | - | - | 85.15 $_{0.28}$ | 75.03 $_{0.85}$ |
| RoBERTa | - | 97.58 $_{0.27}$ | 92.43 $_{0.99}$ | - | - | - | - | 73.88 $_{1.06}$ | 63.28 $_{0.90}$ |
| Rodriguez-Ruiz et al. | - | 78.64 $_{0.00}$ | 79.47 $_{0.00}$ | - | - | - | - | 61.60 $_{0.00}$ | 33.23 $_{0.00}$ |
| Santos et al. | 50.00 $_{0.00}$ | 72.86 $_{0.00}$ | 81.71 $_{0.00}$ | 75.68 $_{0.00}$ | 65.39 $_{0.00}$ | 32.26 $_{0.00}$ | 88.05 $_{0.00}$ | 62.73 $_{0.00}$ | - |
| SATAR | - | 90.66 $_{0.67}$ | - | - | - | - | - | 81.50 $_{1.45}$ | - |
| SGBot | 59.70 $_{3.91}$ | 99.45 $_{0.20}$ | 98.26 $_{0.17}$ | 83.08 $_{2.60}$ | 83.90 $_{0.29}$ | 82.68 $_{1.88}$ | 99.35 $_{0.22}$ | 76.40 $_{0.40}$ | 73.11 $_{0.18}$ |
| T5 | - | 91.04 $_{0.29}$ | 94.48 $_{0.65}$ | - | - | - | - | 72.19 $_{0.84}$ | 63.27 $_{0.71}$ |
| Varol et al. | - | 92.22 $_{0.66}$ | - | - | - | - | - | 78.04 $_{0.61}$ | 75.74 $_{0.31}$ |
| Wei et al. | - | 91.70 $_{1.70}$ | 85.90 $_{1.90}$ | - | - | - | - | 61.00 $_{2.10}$ | 62.70 $_{1.80}$ |
Recall
| Recall | Botometer-feedback-2019 | Cresci-2015 | Cresci-2017 | Cresci-rtbust-2019 | Cresci-stock-2018 | Gilani-2017 | Midterm-2018 | Twibot-20 | Twibot-22 |
|---|---|---|---|---|---|---|---|---|---|
| Abreu et al. | 46.66 $_{3.00}$ | 62.13 $_{0.97}$ | 91.97 $_{0.69}$ | 89.18 $_{1.40}$ | 75.67 $_{0.73}$ | 58.87 $_{2.75}$ | 98.63 $_{0.08}$ | 82.81 $_{0.51}$ | 11.73 $_{0.06}$ |
| Alhosseini et al. | - | 97.16 $_{0.81}$ | - | - | - | - | - | 95.69 $_{1.93}$ | 56.75 $_{17.69}$ |
| BGSRD | 8.57 $_{8.52}$ | 95.56 $_{2.02}$ | 100.0 $_{0.00}$ | 35.14 $_{20.6}$ | 70.40 $_{26.1}$ | 60.00 $_{54.8}$ | 97.66 $_{3.66}$ | 73.19 $_{7.49}$ | 19.90 $_{27.2}$ |
| BotHunter | - | 91.48 $_{4.16}$ | 85.40 $_{0.19}$ | 83.02 $_{2.95}$ | 79.92 $_{0.54}$ | 62.29 $_{3.47}$ | 99.66 $_{0.06}$ | 86.75 $_{0.46}$ | 14.07 $_{0.12}$ |
| Botometer | 57.14 - | 98.95 - | 99.69 - | 100.0 - | 94.96 - | 89.91 - | 87.88 - | 50.82 - | 69.80 - |
| BotRGCN | - | 99.17 $_{0.25}$ | - | - | - | - | - | 90.19 $_{1.72}$ | 46.80 $_{2.76}$ |
| Cresci | - | 66.67 - | 95.30 - | - | - | - | - | 67.47 - | - |
| Dehghan et al. | - | 83.88 $_{0.00}$ | - | - | - | - | - | 82.19 $_{0.00}$ | - |
| Efthimion et al. | 0.00 $_{0.00}$ | 94.38 $_{0.00}$ | 89.23 $_{0.00}$ | 75.68 $_{0.00}$ | 58.02 $_{0.00}$ | 2.80 $_{0.00}$ | 94.04 $_{0.00}$ | 70.63 $_{0.00}$ | 16.76 $_{0.00}$ |
| EvolveBot | - | 95.83 $_{0.66}$ | - | - | - | - | - | 72.81 $_{0.41}$ | 8.04 $_{0.05}$ |
| FriendBot | - | 100.0 $_{0.00}$ | 100.0 $_{0.00}$ | - | - | - | - | 88.94 $_{0.59}$ | - |
| GCN | - | 98.81 $_{0.20}$ | - | - | - | - | - | 87.62 $_{3.31}$ | 44.80 $_{1.71}$ |
| GAT | - | 99.11 $_{0.51}$ | - | - | - | - | - | 89.53 $_{0.87}$ | 44.12 $_{1.65}$ |
| GraphHist | - | 100.0 $_{0.00}$ | - | - | - | - | - | 99.05 $_{0.20}$ | - |
| Hayawi et al. | 17.78 $_{0.06}$ | 79.31 $_{0.02}$ | 92.19 $_{0.03}$ | 81.25 $_{0.09}$ | 71.16 $_{0.07}$ | 28.00 $_{0.13}$ | 98.64 $_{0.00}$ | 83.50 $_{0.04}$ | 14.99 $_{0.05}$ |
| HGT | - | 99.11 $_{0.12}$ | - | - | - | - | - | 91.00 $_{0.57}$ | 28.03 $_{2.60}$ |
| SimpleHGN | - | 99.29 $_{0.40}$ | - | - | - | - | - | 92.06 $_{0.51}$ | 32.90 $_{1.64}$ |
| Kantepe et al. | - | 75.30 $_{1.20}$ | 76.10 $_{1.10}$ | - | - | - | - | 61.00 $_{1.90}$ | 46.80 $_{1.30}$ |
| Knauth et al. | 59.09 $_{0.00}$ | 97.40 $_{0.00}$ | 95.35 $_{0.00}$ | 51.24 $_{0.00}$ | 88.83 $_{0.00}$ | 44.00 $_{0.00}$ | 83.99 $_{0.00}$ | 76.30 $_{0.00}$ | - |
| Kouvela et al. | 20.00 $_{4.71}$ | 96.79 $_{0.75}$ | 98.98 $_{0.18}$ | 80.00 $_{1.48}$ | 78.78 $_{0.18}$ | 57.20 $_{2.42}$ | 98.92 $_{0.06}$ | 95.17 $_{0.14}$ | 19.17 $_{0.04}$ |
| Kudugunta et al. | 45.33 $_{8.69}$ | 60.95 $_{0.21}$ | 85.88 $_{0.37}$ | 50.67 $_{1.21}$ | 47.54 $_{0.60}$ | 35.14 $_{1.70}$ | 90.24 $_{0.66}$ | 33.47 $_{1.30}$ | 61.98 $_{0.00}$ |
| Lee et al. | 44.00 $_{3.65}$ | 98.46 $_{0.14}$ | 99.13 $_{0.00}$ | 86.45 $_{1.44}$ | 80.30 $_{0.63}$ | 60.19 $_{2.15}$ | 98.37 $_{0.10}$ | 83.66 $_{0.69}$ | 19.65 $_{0.15}$ |
| LOBO | - | 99.05 $_{0.13}$ | 96.13 $_{0.39}$ | - | - | - | - | 87.81 $_{0.37}$ | 25.91 $_{0.20}$ |
| Miller et al. | 0.00 $_{0.00}$ | 100.0 $_{0.00}$ | 99.11 $_{0.11}$ | 37.50 $_{0.00}$ | 58.89 $_{0.00}$ | 77.19 $_{0.00}$ | 99.81 $_{0.00}$ | 97.44 $_{0.47}$ | 97.89 $_{0.01}$ |
| Moghaddam et al. | - | 59.23 $_{0.32}$ | - | - | - | - | - | 84.38 $_{1.03}$ | 21.02 $_{0.07}$ |
| NameBot | 33.33 $_{0.00}$ | 91.12 $_{0.00}$ | 91.79 $_{0.00}$ | 70.27 $_{0.00}$ | 64.13 $_{0.00}$ | 36.45 $_{0.00}$ | 96.82 $_{0.00}$ | 70.47 $_{0.00}$ | 0.03 $_{0.00}$ |
| RGT | - | 99.23 $_{0.15}$ | - | - | - | - | - | 91.06 $_{0.80}$ | 30.10 $_{0.17}$ |
| RoBERTa | - | 94.11 $_{0.58}$ | 96.27 $_{1.05}$ | - | - | - | - | 72.38 $_{2.05}$ | 12.27 $_{1.22}$ |
| Rodriguez-Ruiz et al. | - | 99.11 $_{0.00}$ | 92.88 $_{0.00}$ | - | - | - | - | 98.75 $_{0.00}$ | 81.32 $_{0.00}$ |
| Santos et al. | 13.33 $_{0.00}$ | 85.80 $_{0.00}$ | 84.40 $_{0.00}$ | 75.68 $_{0.00}$ | 64.95 $_{0.00}$ | 9.35 $_{0.04}$ | 97.24 $_{0.00}$ | 58.13 $_{0.00}$ | - |
| SATAR | - | 99.88 $_{0.16}$ | - | - | - | - | - | 91.22 $_{1.82}$ | - |
| SGBot | 45.33 $_{2.98}$ | 63.67 $_{1.31}$ | 90.86 $_{0.39}$ | 81.62 $_{2.26}$ | 81.03 $_{0.90}$ | 63.62 $_{2.17}$ | 99.66 $_{0.20}$ | 94.91 $_{0.69}$ | 24.32 $_{0.09}$ |
| T5 | - | 87.71 $_{0.66}$ | 90.26 $_{0.54}$ | - | - | - | - | 69.05 $_{1.46}$ | 12.09 $_{1.43}$ |
| Varol et al. | - | 97.40 $_{0.90}$ | - | - | - | - | - | 84.37 $_{0.67}$ | 16.83 $_{0.21}$ |
| Wei et al. | - | 75.30 $_{1.50}$ | 72.10 $_{1.50}$ | - | - | - | - | 54.00 $_{2.70}$ | 46.80 $_{1.40}$ |
F1
| F1 | Botometer-feedback-2019 | Cresci-2015 | Cresci-2017 | Cresci-rtbust-2019 | Cresci-stock-2018 | Gilani-2017 | Midterm-2018 | Twibot-20 | Twibot-22 |
|---|---|---|---|---|---|---|---|---|---|
| Abreu et al. | 53.84 $_{3.03}$ | 76.36 $_{0.72}$ | 95.04 $_{0.30}$ | 83.54 $_{1.04}$ | 76.93 $_{0.58}$ | 66.66 $_{0.10}$ | 97.95 $_{0.03}$ | 77.14 $_{0.46}$ | 53.44 $_{0.09}$ |
| Alhosseini et al. | - | 92.17 $_{0.36}$ | - | - | - | - | - | 72.07 $_{0.48}$ | 38.10 $_{5.93}$ |
| BGSRD | 13.03 $_{13.0}$ | 90.80 $_{0.60}$ | 86.27 $_{0.00}$ | 41.08 $_{13.0}$ | 58.18 $_{12.1}$ | 35.72 $_{32.6}$ | 90.50 $_{1.09}$ | 70.05 $_{2.60}$ | 21.14 $_{29.0}$ |
| BotHunter | 49.57 $_{3.12}$ | 97.22 $_{0.96}$ | 91.60 $_{3.12}$ | 82.90 $_{1.88}$ | 82.17 $_{0.20}$ | 69.18 $_{1.04}$ | 99.59 $_{0.02}$ | 79.09 $_{0.36}$ | 23.46 $_{0.09}$ |
| Botometer | 30.77 - | 66.90 - | 96.12 - | 78.95 - | 79.59 - | 77.39 - | 46.03 - | 53.13 - | 42.75 - |
| BotRGCN | - | 97.30 $_{0.53}$ | - | - | - | - | - | 87.25 $_{0.73}$ | 57.50 $_{1.42}$ |
| Cresci | - | 1.17 - | 22.81 - | - | - | - | - | 13.69 - | - |
| Dehgan | - | 88.34 $_{0.00}$ | - | - | - | - | - | 76.20 $_{0.00}$ | - |
| Efthimion et al. | 0.00 $_{0.00}$ | 94.10 $_{0.00}$ | 91.83 $_{0.00}$ | 71.79 $_{0.00}$ | 68.21 $_{0.00}$ | 05.22 $_{0.00}$ | 95.98 $_{0.00}$ | 67.26 $_{0.00}$ | 27.58 $_{0.00}$ |
| EvolveBot | - | 90.07 $_{1.98}$ | - | - | - | - | - | 69.75 $_{0.50}$ | 14.09 $_{0.08}$ |
| FriendBot | - | 97.58 $_{0.84}$ | 87.35 $_{0.52}$ | - | - | - | - | 79.97 $_{0.34}$ | - |
| GCN | - | 97.17 $_{0.43}$ | - | - | - | - | - | 80.86 $_{0.68}$ | 54.96 $_{0.91}$ |
| GAT | - | 97.58 $_{0.15}$ | - | - | - | - | - | 85.25 $_{0.38}$ | 55.86 $_{1.01}$ |
| GraphHist | - | 84.47 $_{8.23}$ | - | - | - | - | - | 67.56 $_{0.30}$ | - |
| Hayawi et al. | 20.49 $_{0.06}$ | 85.56 $_{0.01}$ | 93.78 $_{0.01}$ | 60.87 $_{0.03}$ | 60.75 $_{0.06}$ | 34.67 $_{0.11}$ | 91.48 $_{0.00}$ | 77.05 $_{0.02}$ | 24.74 $_{0.08}$ |
| HGT | - | 96.93 $_{0.24}$ | - | - | - | - | - | 88.19 $_{0.19}$ | 39.60 $_{2.11}$ |
| SimpleHGN | - | 97.28 $_{0.39}$ | - | - | - | - | - | 88.25 $_{0.18}$ | 45.44 $_{1.65}$ |
| Kantepe et al. | - | 78.17 $_{1.42}$ | 79.41 $_{1.27}$ | - | - | - | - | 62.23 $_{2.06}$ | 58.71 $_{1.61}$ |
| Knauth et al. | 41.27 $_{0.00}$ | 91.18 $_{0.00}$ | 93.42 $_{0.00}$ | 54.15 $_{0.00}$ | 94.03 $_{0.00}$ | 39.10 $_{0.00}$ | 91.26 $_{0.00}$ | 85.24 $_{0.00}$ | 37.09 $_{0.00}$ |
| Kouvela et al. | 28.10 $_{5.27}$ | 98.15 $_{0.38}$ | 99.11 $_{0.06}$ | 81.10 $_{1.03}$ | 80.44 $_{0.23}$ | 66.57 $_{1.72}$ | 98.23 $_{0.05}$ | 86.53 $_{0.26}$ | 30.03 $_{0.04}$ |
| Kudugunta et al. | 49.61 $_{8.20}$ | 75.74 $_{0.16}$ | 91.74 $_{0.17}$ | 49.22 $_{1.28}$ | 50.94 $_{0.38}$ | 49.75 $_{2.10}$ | 94.45 $_{0.32}$ | 47.26 $_{1.35}$ | 51.67 $_{0.00}$ |
| Lee et al. | 50.34 $_{3.16}$ | 98.56 $_{0.11}$ | 99.35 $_{0.04}$ | 82.74 $_{1.79}$ | 82.46 $_{0.36}$ | 67.78 $_{1.81}$ | 97.87 $_{0.07}$ | 79.98 $_{0.50}$ | 30.41 $_{0.20}$ |
| LOBO | - | 98.76 $_{0.26}$ | 97.69 $_{0.18}$ | - | - | - | - | 80.80 $_{0.20}$ | 38.57 $_{0.23}$ |
| Miller et al. | 0.00 $_{0.00}$ | 83.77 $_{0.00}$ | 86.80 $_{0.07}$ | 43.64 $_{0.00}$ | 56.76 $_{0.00}$ | 59.86 $_{0.00}$ | 91.14 $_{0.00}$ | 74.81 $_{0.26}$ | 45.29 $_{0.00}$ |
| Moghaddam et al. | - | 73.93 $_{0.21}$ | - | - | - | - | - | 77.87 $_{0.71}$ | 32.07 $_{0.03}$ |
| NameBot | 38.46 $_{0.00}$ | 83.36 $_{0.00}$ | 85.71 $_{0.02}$ | 67.53 $_{0.00}$ | 61.10 $_{0.00}$ | 44.83 $_{0.00}$ | 91.61 $_{0.00}$ | 65.06 $_{0.00}$ | 0.50 $_{0.00}$ |
| RGT | - | 97.78 $_{0.24}$ | - | - | - | - | - | 88.01 $_{0.41}$ | 42.94 $_{1.85}$ |
| RoBERTa | - | 95.86 $_{0.19}$ | 94.30 $_{0.18}$ | - | - | - | - | 73.09 $_{0.59}$ | 20.53 $_{1.71}$ |
| Rodriguez-Ruiz et al. | - | 87.70 $_{0.00}$ | 85.65 $_{0.00}$ | - | - | - | - | 63.10 $_{0.00}$ | 56.57 $_{0.00}$ |
| Santos et al. | 21.05 $_{0.00}$ | 78.80 $_{0.00}$ | 83.03 $_{0.00}$ | 75.68 $_{0.00}$ | 65.17 $_{0.00}$ | 14.49 $_{0.00}$ | 92.42 $_{0.00}$ | 60.34 $_{0.00}$ | - |
| SATAR | - | 95.05 $_{0.34}$ | - | - | - | - | - | 86.07 $_{0.70}$ | - |
| SGBot | 49.60 $_{3.43}$ | 77.91 $_{0.13}$ | 94.61 $_{0.19}$ | 82.26 $_{1.73}$ | 82.34 $_{0.11}$ | 72.10 $_{0.19}$ | 99.52 $_{0.02}$ | 84.90 $_{0.42}$ | 36.59 $_{0.18}$ |
| T5 | - | 89.35 $_{0.26}$ | 92.32 $_{0.11}$ | - | - | - | - | 70.57 $_{0.39}$ | 20.27 $_{2.03}$ |
| Varol et al. | - | 94.73 $_{0.42}$ | - | - | - | - | - | 81.08 $_{0.48}$ | 27.54 $_{0.26}$ |
| Wei et al. | - | 82.65 $_{2.21}$ | 78.43 $_{1.66}$ | - | - | - | - | 57.33 $_{3.19}$ | 53.61 $_{1.36}$ |
Test1
| model | Acc | F1 | precision | recall |
|---|---|---|---|---|
| Moghaddam et al. | 89.41 $_{0.30}$ | 24.98 $_{2.72}$ | 16.57 $_{1.97}$ | 50.79 $_{4.25}$ |
| SGBot | 91.87 $_{0.11}$ | 47.43 $_{1.21}$ | 76.16 $_{2.31}$ | 34.48 $_{1.56}$ |
| BotHunter | 91.44 $_{0.12}$ | 40.39 $_{0.32}$ | 78.28 $_{3.11}$ | 27.24 $_{0.52}$ |
| GAT | 91.14 $_{0.45}$ | 47.00 $_{2.92}$ | 64.83 $_{4.31}$ | 36.95 $_{3.04}$ |
| BotRGCN | 88.74 $_{0.29}$ | 65.89 $_{1.62}$ | 79.82 $_{2.53}$ | 56.23 $_{3.24}$ |
| RGT | 92.8 $_{0.45}$ | 23.39 $_{4.61}$ | 58.33 $_{11.78}$ | 14.66 $_{2.98}$ |
Test2
| model | Acc | F1 | precision | recall |
|---|---|---|---|---|
| Moghaddam et al. | 83.93 $_{0.28}$ | 18.49 $_{0.95}$ | 11.58 $_{0.59}$ | 45.94 $_{3.35}$ |
| SGBot | 84.72 $_{0.31}$ | 26.00 $_{2.80}$ | 54.55 $_{2.80}$ | 17.11 $_{2.28}$ |
| BotHunter | 85.63 $_{0.31}$ | 23.38 $_{1.55}$ | 73.67 $_{9.81}$ | 13.95 $_{1.18}$ |
| GAT | 84.93 $_{0.23}$ | 30.47 $_{2.64}$ | 55.64 $_{2.02}$ | 21.05 $_{2.46}$ |
| BotRGCN | 85.59 $_{0.68}$ | 55.45 $_{2.77}$ | 67.45 $_{2.74}$ | 47.17 $_{3.65}$ |
| RGT | 87.1 $_{1.19}$ | 38.02 $_{7.21}$ | 58.50 $_{10.18}$ | 28.57 $_{6.68}$ |
Test3
| model | Acc | F1 | precision | recall |
|---|---|---|---|---|
| Moghaddam et al. | 87.61 $_{0.20}$ | 22.34 $_{1.78}$ | 14.48 $_{1.26}$ | 49.00 $_{2.74}$ |
| SGBot | 89.52 $_{0.13}$ | 38.96 $_{1.77}$ | 68.97 $_{1.57}$ | 27.18 $_{1.85}$ |
| BotHunter | 89.53 $_{0.12}$ | 33.77 $_{0.45}$ | 76.62 $_{2.45}$ | 21.66 $_{0.25}$ |
| GAT | 89.09 $_{0.38}$ | 40.58 $_{2.68}$ | 61.84 $_{3.50}$ | 30.28 $_{2.75}$ |
| BotRGCN | 87.92 $_{0.51}$ | 59.46 $_{2.36}$ | 76.88 $_{3.71}$ | 48.66 $_{3.76}$ |
| RGT | 89.6 $_{0.72}$ | 26.89 $_{4.71}$ | 56.49 $_{11.34}$ | 18.05 $_{3.63}$ |
Citation
Please cite TwiBot-22 if you use the TwiBot-22 dataset or this repository
@inproceedings{fengtwibot,
title={TwiBot-22: Towards Graph-Based Twitter Bot Detection},
author={Feng, Shangbin and Tan, Zhaoxuan and Wan, Herun and Wang, Ningnan and Chen, Zilong and Zhang, Binchi and Zheng, Qinghua and Zhang, Wenqian and Lei, Zhenyu and Yang, Shujie and others},
booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track}
}How to contribute
- New dataset: convert the original data to the TwiBot-22 defined schema.
- New baseline: load well-formatted dataset from the dataset directory and define your model.
Welcome PR!
Questions?
Feel free to open issues in this repository! Instead of emails, Github issues are much better at facilitating a conversation between you and our team to address your needs. You can also contact Zhaoxuan Tan through tanzhaoxuan at stu.xjtu.edu.cn.