The FATE-Operator makes it easy to deploy and run federated machine learning (FML) jobs on Kubernetes. It is a early version, all suggestions and feature requests are valueables for us. Raising issues is appreciate.
Federated machine learning (FML) is a machine learning setting where many clients (e.g. mobile devices or organizations) collaboratively train a model under the coordination of a central server while keeping the training data decentralized. Only the encrypted mediate parameters are exchanged between clients with MPC or homomorphic encryption.
FML has received significant interest recently, because of its effectiveness to solve data silos and data privacy preserving problems. Companies participated in federated machine learning include 4Paradigm, Ant Financial, Data Republic, Google, Huawei, Intel, JD.com, Microsoft, Nvidia, OpenMind, Pingan Technology, Sharemind, Tencent, VMware, Webank etc.
Depending on the differences in features and sample data space, federated machine learning can be classified into horizontal federated machine learning, vertical federated machine learning and federated transfer learning. Horizontal federated machine learning is also called sample-based federated machine learning, which means data sets share the same feature space but have different samples. With horizontal federated machine learning, we can gather the relatively small or partial datasets into a big one to increase the performance of trained models. Vertical federated machine learning is applicable to the cases where there are two datasets with different feature space but share same sample ID. With vertical federated machine learning we can train a model with attributes from different organizations for a full profile. Vertical federated machine learning is required to redesign most of machine learning algorithms. Federated transfer learning applies to scenarios where there are two datasets with different features space but also different samples.
FATE (Federated AI Technology Enabler) is an open source project initialized by Webank, now hosted at the Linux Foundation. FATE is the only open source FML framework that supports both horizontal and vertical FML currently. The architecture design of FATE is focused on providing FML platform for enterprises. KubeFATE is an open source project to deploy FATE on Kubernetes and is a proven effective solution for FML use cases.
More technologies of Federated machine learning, please refer to Reference section
The design proposal of fate-operator can be found in kubeflow/community/fate-operator-proposal.md.
The fate-operator can be installed by following steps.
make install
make uninstall
The Docker images are built and pushed to Dockerhub. fate-operator
Alternatively, we can build the images manually by commands,
make docker-build
make deploy
KubeFATE is the infrastructure management service for multiple FATE clusters in one organization. It will only deploy once. To deploy a KubeFATE service, we use the YAML refer to app_v1beta1_kubefate.yaml as an example,
kubectl create -f ./config/samples/app_v1beta1_kubefate.yaml
FATE is the cluster we run FML jobs. To deploy a FATE cluster, we use YAML refer to app_v1beta1_fatecluster.yaml as an example,
kubectl create -f ./config/samples/app_v1beta1_fatecluster.yaml
Once KubeFATE and FATE cluster deployed, we can submit a FML job with FateJob
config file. In current version, the FATE Job is defined by DSL pipeline
and Module Config
two parts. The details refer to app_v1beta1_fatejob.yaml.
kubectl create -f ./config/samples/app_v1beta1_fatejob.yaml
In this example, only a secure add
operation will be processed. For more example, we can refer to FATE Examples. In each example, the files end with _dsl
, e.g. https://github.com/FederatedAI/FATE/blob/master/examples/federatedml-1.x-examples/hetero_linear_regression/test_hetero_linr_cv_job_dsl.json is the job pipeline and what should be put in pipeline
field; and the files end with _conf
, e.g. https://github.com/FederatedAI/FATE/blob/master/examples/federatedml-1.x-examples/hetero_linear_regression/test_hetero_linr_cv_job_conf.json is the config of each components and what should be put in modulesConf
field.
The status of created resource can be monitor with kubectl get
command,
kubectl get kubefate,fatecluster,fatejob -A -o yaml