Pause a consumer for X minutes

[jwreiser]

Expected Behavior

@Component @KafkaListener(topics = {"job.cache"}, groupId = "cache-consumer") public class JobCacheConsumer { private static Cache cache = new MongoDbCache();

@KafkaHandler
@RetryableTopic(
        backoff = @Backoff(delay = 10_000, multiplier = 2, maxDelay = 36_000_000),
        attempts = "-1",
        autoCreateTopics = "true",
        include = SocketTimeoutException.class, exclude = CachRetryException.class)
public void jobHandler(Job job, @Headers Map<String, Object> headers
        , @Header(KafkaHeaders.RECEIVED_KEY) String key
                       ) throws CachRetryException {

try{ callForeignResource() }catch(SocketTimeoutException){ //PAUSE FOR 30 minutes }

Current Behavior

As far as I can tell you can only pause and resume. I am not sure how to get pausing to be time constrained

Context

I am calling mongodb (in the cloud) when I process certain topics. If mongo is down I don't want to continue processing events in the queue as they are all likely to fail unnecessarily consuming resources. I'd like to pause the whole topic while I give mongo some time to fix itself.

[sobychacko]

@jwreiser Why can't you just programmatically pause and then resume? (as documented here).

[jwreiser]

@jwreiser Why can't you just programmatically pause and then resume? (as documented here). Thank you for the quick response.

For three reasons.

1) I don't want to sleep a thread as the example does. I imagine I could figure out how to use a scheduler or something like that, but I might get bits wrong which is why I think based on 3 a reusable solution would be useful. 2) The example is also quite different from my setup now and I don't understand how to integrate it into my setup. I haven't found any good books or courses on Spring Kafka so I don't have a good overall view yet. 3) I would think lots of people would want this. Any time something is down we don't want to keep hammering it and further cripple it.

[sobychacko]

Yes, so this use case of pausing/resuming the container/consumer is very common in enterprise applications. Usually, the users are fine with what is available currently, i.e., pausing programmatically until a condition is met and then resuming. What kind of solutions are you thinking about/proposing here? If you have ideas for improving it, we are all ears. Thanks!

[sobychacko]

@jwreiser Any updates on this issue?

[sobychacko]

Closing the issue due to no activity. Feel free to re-open if there is a need. Thanks!