ElasticFlow implementation tasks

[zou000]

ParameterServer

1. RPC interface

   • [ ] service definition
   • [ ] SparseTensor support

2. RPC implementation

   • [ ] TF/PyTorch tensor to.from Tensor proto bidirectional convertors
   • [ ] multi-threaded Server
   • [ ] Client

3. PS implementation

   • Graph gradient update
   • large model support: model partition
   • large lookup table support (e.g. use redis)

Data API

1. data fetcher: Need a unified way for feeding data.
   • ODPS support for TF, PyTorch
     • tf.data.DataSet and torch.utils.data.Dataset
   • Other data source?
2. data sharding
   • sharding methods
   • shard tracking: re-queue shard when worker dies
     • use etcd?

Worker with PS client/data fetcher/etcd client

1. PyTorch worker
2. Estimator Worker
   • Current POC crashes when iteration number is high, need to debug first
3. Other TF workers

Master

• divide work, wait for workers to finish

Job launcher

1. based on user code, decide PS and worker to launch
2. local launcher (for testing)
3. kubernetes launcher
   • on premise
   • AWS
   • GKE

Elasticity, fault tolerance support

• periodic checkpoint/model saving (to file? redis?)
• initial model loading (from file? redis? PS?)
• PS scaling
  • partitioned model support

Kubernetes controller

1. resource tracking
2. scaling

[wangkuiyi]

Thanks to @zou000 for this plan!

A few cents from me:

1. For offline training, ElasticFlow needs to be able to read from files on (distributed) filesystems. The file format must support partition in order to enable distributed training. More about the motivation is in https://github.com/wangkuiyi/recordio/blob/master/README.md.