search

ethanabrooks / computational-graph

Efficiently performs automatic differentiation on arbitrary functions. Basically a rudimentary version of Tensorflow.
12 stars 1 forks source link

Similar project #2

Open botev opened 7 years ago

botev commented 7 years ago

Hi,

So we had similar start of a similar project here: https://github.com/Metadiff/gir. We occasionally chat in gitter on https://gitter.im/rust-ml/ if by any chance you are interested in potentially putting efforts together or just general discussion about ML and graph computation optimizers please feel free to come and have a chat.

ethanabrooks commented 7 years ago

Hi Alexander! Thanks for your interest. I think it makes sense to combine our efforts. What did you have in mind?

On Sun, Feb 12, 2017 at 5:14 PM Alexander Botev notifications@github.com wrote:

Hi,

So we had similar start of a similar project here: https://github.com/Metadiff/gir. We occasionally chat in gitter on https://gitter.im/rust-ml/ if by any chance you are interested in potentially putting efforts together or just general discussion about ML and graph computation optimizers please feel free to come and have a chat.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lobachevzky/computational-graph/issues/2, or mute the thread https://github.com/notifications/unsubscribe-auth/AJ3ZJuLhkFWU2cA8jSJYyKj0-3rNGZpjks5rb4RVgaJpZM4L-pEk .

botev commented 7 years ago

So in general, I think putting efforts together is really good. However, I know that it is possible that for instance maybe you do not agree with my approach etc.. which why the discussion is good as we can talk about good and bad sides.

Generally speaking, I think the main difference is that I do not build the graph in memory (e.g. as enum tree), but it is a fully dynamic structure. I think that is better as it allows for easier access to both parents and child, and also for later down the pipeline inserting auxiliary nodes (for instance fork and join or barriers). It also I think makes it easier to modify the graph during optimization procedure. Additionally, modeling it as enum prevents downstream backends to create their own ops, for instance a cuda + cudnn might want to make its adhoc cudnn_conv2d operator which to be inserted instead of the general conv2d etc...

Second, I explicitly model the shapes of the tensors and their data types, to allow shape verification for dynamic shapes even at graph construction type and other such checks. This really gives significantly better debugging messages than anything else out there. It also requires a bit more verbosity, but it fits well I think in the Rust spirit.

Another main goal, at least for me, is to make the graph engine, totally serialiazable and separate from the backends. This roughly follows the LLVM paradigm of allowing an independent IR, which can be reused, and which later each backend can execute as it pleases.

At the moment, I have a very limited example, with backward differentiation implemented. Currently, I'm plugging in Arrayfire as the easiest initial backend to use (its the least work of the rest). I also want to focus a lot on OpenCL as well as CUDA, but my experience there is not that great. In general I would like to use template engines like tera to be able to generate ad-hoc kernels based on the graph.

Currently, any help in any of the departments is more than welcome. Any suggestions, critiques etc of design choices are also most welcome and any discussion on those. I mostly tried to pick the best things from Theano, Tf and the other frameworks and try to make something like an LLVM for this. Main point being that a lot of the optimization can happen in the IR, such as memory allocation and scheduling, and this can be reused by anyone using it, independent of the backend.

I do not know what are your goals/aims out of this as well as what are your main strengths in this kind of area. Again feel free to drop to the gitter channel.