tumblr / collins

groovy kind of love
tumblr.github.com/collins
Apache License 2.0
571 stars 99 forks source link

Implement a firehose for events #379

Closed yl3w closed 8 years ago

yl3w commented 8 years ago

Summary: A firehose implementation using server sent events. Relies on callback mechanism to implement the firehose. All events from the firehose return as part of data the asset that is associated with the event.

All events for assets, asset meta values and ip addresses are supported.

Every event includes an event name. Supports clustered installations by making use of hazelcast reliable topic.

The firehose callback processing actor writes events to the hazelcast reliable topic. We rely on hazelcast to distribute the event to the cluster.

A single instance of collins installs a listener for the topic, receives the message and writes it out to all clients of the firehose.

Prerequisites for using the firehose:

Hazelcast MUST be enabled, does NOT need to be used for cache. This deployment is applicable when running in stand alone more.

Callbacks MUST be enabled. Callbacks are also required for search (Solr).

Configuration:

Hazelcast configuration is captured in conf/hazelcast.xml. The topic can documentation is at [1].

[1] http://docs.hazelcast.org/docs/latest-dev/manual/html/reliabletopicconfiguration.html#reliable-topic-configuration

yl3w commented 8 years ago

@roymarantz @Primer42 @byxorna @defect

I'm going to do the tests separately.