Use synchronous queue when sink does not want to drop message to deal with back pressure

allenxwang commented 9 years ago

ElasticSearchSink uses MessageQueue4Sink for message handoff between the input and the sink. Its writeTo() method is simply implemented as enqueuing the message to MessageQueue4Sink, which will drop the message if queue is full. Therefore, the queue for ElasticSearchSink is typically configured as a large file based queue to minimize the possibility of dropping messages. This complicates some Suro operations and monitoring as it will have local storage and hence becomes stateful.

In case that the input to ElasticSearchSink is KafkaConsumer, where there is already a file queue offered by Kafka, having another queue in the system makes things complicated and unnecessary. In this case, if sink is too busy, it should just naturally slow down the input (KafkaConsumer) which will cause lag in the consumer but without having to deal with back pressure.

Therefore, a synchronous queue seems to be a good fit in this case where it can block the input if sink busy, and shift the back pressure to its upstream.

With this update, user can configure the queue to be synchronous queue by specifying the capacity of the queue to be 0.