Closed allenxwang closed 9 years ago
NetflixOSS » suro » suro-pull-requests #60 SUCCESS This pull request looks good
suro-pull-requests #228 FAILURE Looks like there's a problem with this pull request
we should make this change in the master branch. thrift-0.7.0 is only used for platform logging. suro server/router uses the maser branch.
ElasticSearchSink
usesMessageQueue4Sink
for message handoff between the input and the sink. ItswriteTo()
method is simply implemented as enqueuing the message toMessageQueue4Sink
, which will drop the message if queue is full. Therefore, the queue forElasticSearchSink
is typically configured as a large file based queue to minimize the possibility of dropping messages. This complicates some Suro operations and monitoring as it will have local storage and hence becomes stateful.In case that the input to
ElasticSearchSink
isKafkaConsumer
, where there is already a file queue offered by Kafka, having another queue in the system makes things complicated and unnecessary. In this case, if sink is too busy, it should just naturally slow down the input (KafkaConsumer
) which will cause lag in the consumer but without having to deal with back pressure.Therefore, a synchronous queue seems to be a good fit in this case where it can block the input if sink busy, and shift the back pressure to its upstream.
With this update, user can configure the queue to be synchronous queue by specifying the capacity of the queue to be 0.