πFor the benchmark resultπ, please check our website.πππ
Scenario name | Modality | Task type | Performance metrics | Client number | Sample number |
---|---|---|---|---|---|
celeba | Image | Binary Classification (Smiling vs. Not smiling) |
Accuracy | 894 | 20,028 |
femnist | Image | Multiclass Classification (62 classes) |
Accuracy | 178 | 40,203 |
Text | Next-word Prediction | Accuracy | 813 | 27,738 |
Scenario name | Modality | Task type | Performance metrics | Client number | Sample number |
---|---|---|---|---|---|
breast_horizontal | Medical | Binary Classification | AUC | 2 | 569 |
default_credit_horizontal | Tabular | Binary Classification | AUC | 2 | 22,000 |
give_credit_horizontal | Tabular | Binary Classification | AUC | 2 | 150,000 |
student_horizontal | Tabular | Regression (Grade Estimation) |
MSE | 2 | 395 |
vehicle_scale_horizontal | Image | Multiclass Classification (4 classes) |
Accuracy | 2 | 846 |
Scenario name | Modality | Task type | Performance metrics | Vertical split details |
---|---|---|---|---|
breast_vertical | Medical | Binary Classification | AUC | A: 10 features 1 label B: 20 features |
default_credit_vertical | Tabular | Binary Classification | AUC | A: 13 features 1 label B: 10 features |
dvisits_vertical | Tabular | Regression (Number of consultations Estimation) |
MSE | A: 3 features 1 label B: 9 features |
give_credit_vertical | Tabular | Binary Classification | AUC | A: 5 features 1 label B: 5 features |
motor_vertical | Sensor data | Regression (Temperature Estimation) |
MSE | A: 4 features 1 label B: 7 features |
student_vertical | Tabular | Regression (Grade Estimation) |
MSE | A: 6 features 1 label B: 7 features |
vehicle_scale_vertical | Image | Multiclass Classification (4 classes) |
Accuracy | A: 9 features 1 label B: 9 features |
Python 3.9
. It should also work on Python>=3.6
, feel free to contact us if you encounter any problems.conda create -n flbenchmark python=3.9
git
, wget
, unzip
pip install flbenchmark colink
We highly recommend to use cloud servers(e.g. AWS) to run the benchmark. Here we provide an auto-deploy script on AWS.
CONTROLLER_URL="http://172.31.2.2"
in install_auto.sh to the controller's web url.http://172.31.2.2
with the controller's web url)
#!/bin/bash
sudo -i -u ubuntu << EOF
echo "Hello World" > testfile.txt
wget http://172.31.2.2/install_auto.sh
bash install_auto.sh > user_data.out
EOF
python get_server_list.py
, and copy the server_list.json
to exp folder.python register_users.py
../start_po.a <unifed_protocol_name> <repo_url>
. For example, ./start_po.a unifed.flower https://github.com/CoLearn-Dev/colink-unifed-flower.git
.python run_task_auto.py
Alternatively, you could also manually deploy on your own cluster. You need to prepare enough Ubuntu 20.04 LTS servers based on the needs of the benchmark, and you should set up the environment on these servers. Here we provide one script to set up the environment.
SERVER_IP="172.16.1.1"
in install.sh to corresponding server ip.~/server/host_token.txt
in the following json format.
{
"test-0": [
"http://172.31.4.48:80",
"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJwcml2aWxlZ2UiOiJob3N0IiwidXNlcl9pZCI6IjAyMGJhNzkyNzk0ZTlmMWUwZWZmNTEyOGM4NDdjZmE0MmRlNTllY2I1ODM4MzU4MDBmN2QwMzM1Yzg2YWFjZTViOSIsImV4cCI6MTY4OTM0ODAyM30.UP5JUYdbL-MkZDTSVuBHnIHoun1VkfcRgsBLV119v6A"
],
"test-1": [
"http://172.31.15.143:80",
"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJwcml2aWxlZ2UiOiJob3N0IiwidXNlcl9pZCI6IjAyNzJhNjgxNDg0NDMwNTFmZTI2NDFlYmZiNjM2MDgxODM5YmQ5NDdkZGFhNTcwYjY3MjU0MTI2NjU5YzBmZjVjYSIsImV4cCI6MTY4OTM0ODAzNH0.2HHQYzcMjif0ZkhSyltlOSC1ydsgWS8H_no5wWvohw0"
]
}
python register_users.py
and record the user id for later use../start_po.a <unifed_protocol_name> <repo_url>
. For example, ./start_po.a unifed.crypten https://github.com/CoLearn-Dev/colink-unifed-crypten.git
.user_id
with the user id you got from the previous steps.python run_task.py
This project is licensed under Apache License Version 2.0. By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms.