search

thinkaurelius / faunus

Graph Analytics Engine
http://faunus.thinkaurelius.com
Apache License 2.0
262 stars 58 forks source link

Make use of single machine in-memory components (Fulgora/TinkerGraph-style) #62

Closed okram closed 11 years ago

okram commented 11 years ago

For some partitioning strategy, if each subgraph/component fits nicely on one machine (to some extent), we could have the mapper load that machine's subgraph into an in-memory graph system (e.g. Fulgrora or TinkerGraph -- or any Graph for that matter), do iterations in memory. Next, once a certain number of iterations have been done in memory and cross component communication is needed, then the reduce phase can propagate the information and write the entire graph back to HDFS. This way, you can get more iterations done in fewer MapReduce iterations.

This is like a hybrid between MapReduce/Pregel/Cassovary.

okram commented 11 years ago

This is good when we start to think of graph contractions (a subgraph as a single vertex).

okram commented 11 years ago

See script() step. Done.