Better Broadcasting Strategies

amplab / velox-modelserver

http://amplab.github.io/velox-modelserver/

Apache License 2.0

110 stars 26 forks source link

Better Broadcasting Strategies #46

Open tomerk opened 9 years ago

tomerk commented 9 years ago

The current broadcasting strategy is to just write into etcd and then read it back from all nodes. Etcd isn't designed for this and this is probably bad for anything over a megabyte in size or so.

Strategies to introduce:

Spark filesystem (kinda like how the observations and user weights reading & writing currently work)
Torrent

This issue also involves decoupling the Model code from the exact type of broadcast being provided.

tomerk commented 9 years ago

As of Issue #51 Broadcast via the spark filesystem exists. In the future it would be nice to have broadcast via HTTP, and broadcast via Torrent