search

jpatanooga / Metronome

Suite of parallel iterative algorithms built on top of Iterative Reduce
Apache License 2.0
106 stars 18 forks source link

examples? #3

Open nouramoubayed opened 9 years ago

nouramoubayed commented 9 years ago

are there any examples of using DL4J with Hadoop? thanks

jpatanooga commented 9 years ago

So Metronome was an earlier parallel iterative suite of algorithms that I did on Hadoop/YARN.

When I started working on DL with Adam back in 2013, initially he wrote a single process version of DL and I wrote a version that worked on Mahout's linear algebra lib that was packaged inside Metronome.

As we moved forward it made more sense to create a larger package due to interest on a unified architecture. So we ported just the parallel DL version from Metronome over into what became DL4J. Under that design we ported the parallel iterative pattern onto the spark version in DL4J as well.

We're currently working out a Command Line Interface (CLI) for DL4J that pulls together the mechanics to use DL4J on different runtimes based on just changing the config file and not writing any code.

There are some unit tests in DL4J currently that show it running on Hadoop clusters.

Hope these notes help,

JP

On Mon, Feb 16, 2015 at 11:43 AM, Nora notifications@github.com wrote:

are there any examples of using DL4J with Hadoop? thanks

— Reply to this email directly or view it on GitHub https://github.com/jpatanooga/Metronome/issues/3.