search

cmu-db / peloton

The Self-Driving Database Management System
http://pelotondb.io
Apache License 2.0
2.04k stars 623 forks source link

Integrate Tensorflow into Peloton #836

Open linmagit opened 7 years ago

linmagit commented 7 years ago

The workload prediction module of our self-driving component uses machine learning to predict the arrival rate time series of queries. We're going to integrate Tensorflow to train those machine learning models. Furthermore, the feedback control part of the self-driving component also needs it to train statistical models. The C++ version of Tensorflow requires us to build tensorflow by ourselves. That will significantly increase Peloton's build time. So for now we have two other choices:

  1. Use Python version of tensorflow in another process (probably in a Docker in the future). Then we open up a pipe between Peloton's main process and this process to communicate.
  2. Use C version of tensorflow: https://www.tensorflow.org/install/install_c It seems that this thing can be directly installed as a library. However, I don't know how compatible this is with C++. Also, the official Tensorflow API documentation provides C++ doc but no C doc: https://www.tensorflow.org/api_docs/ We may need to look further into this.
poojanilangekar commented 6 years ago

@saatviks Are you working on this?

saatviks commented 6 years ago

Yep - Have been coordinating with @malin1993ml. We have finished setting up a helper/entity C++ class to work with the Tensorflow C API which does the following:

  1. Import a tensorflow graph which would generally be written in the tensorflow python API.
  2. Useful methods to setup and call tensorflow ops - these would generally be to train and test models
  3. Examples for vanilla logistic regression and MLP Next up we are working on reproducing @malin1993ml's python code for workload prediction. Will keep this space updated as we finish adding more features.
poojanilangekar commented 6 years ago

@saatviks Thanks for the update. I have assigned you to this issue. Please make sure you mention this issue when you raise a PR.

saatviks commented 6 years ago

Sure, thanks.