Adds support for retraining models in batch using Spark - Githubissues

amplab / velox-modelserver

http://amplab.github.io/velox-modelserver/

Apache License 2.0

110 stars 26 forks source link

Adds support for retraining models in batch using Spark #27

Closed dcrankshaw closed 9 years ago

dcrankshaw commented 9 years ago

This contains the initial skeleton of support needed to retrain models in Spark and then incorporate them into a running Velox system. It adds

global coordination (locking) via Etcd
reads and writes from HDFS as the interchange medium. This is just a shim until both Velox and Spark 1.3 support higher-functionality storage backend (e.g. Cassandra).
Runs a spark program remotely on an existing Spark cluster

While this PR reads the new models back into Velox, it then throws them away rather than using them to serve. This should be fixed in a future PR.