Run Coordinator with Hadoop+YARN

viatra / incqueryd

IncQuery-D: a scalable, distributed incremental graph pattern matching engine

6 stars 1 forks source link

Run Coordinator with Hadoop+YARN #55

Closed thSoft closed 9 years ago

thSoft commented 9 years ago

The Coordinator (and later the Rete actor containers) should be allocated with YARN's Resource Manager & Application Master. This makes Infrastructure Agent unnecessary (and simplifies the Coordinator's logic). The actor service JAR should be accessed from HDFS.

thSoft commented 9 years ago

TODO:

solve hadoop fs -get "No such file or directory"
consolidate constants in scripts & sources
change/remove demo-specific parts

thSoft commented 9 years ago

Next task: When the AM has started the Actor Service Server, send back the IP address of its container's node to the caller. Do this via ZooKeeper.

thSoft commented 9 years ago

Fixed in issues/55 branch. Deferring merging into master until #57 is fixed.