crowdrec / idomaar

CrowdRec reference framework
Apache License 2.0
32 stars 12 forks source link

Idomaar development setup #49

Closed andras-sereny closed 7 years ago

andras-sereny commented 9 years ago

While using Kafka/Flume to stream data makes Idomaar capable of handling high-rate data flows, having to use all these components during development can be troublesome (slow startup, even components not affected by current changes are involved). It'd be nice to be able to develop Idomaar in a simplified setup.

davidemalagoli commented 9 years ago

Perhaps using kafka as an embedded process (https://ransilberman.wordpress.com/2013/07/19/how-to-unit-test-kafka/) and flume as an embedded agent (http://flume.apache.org/releases/content/1.4.0/apidocs/org/apache/flume/agent/embedded/EmbeddedAgent.html) could be used to simulate the "real" ones?