Shutting down consumer task hangs for some time when kafka is unreacheable

[slinkydeveloper]

This is an upstream bug: https://github.com/fede1024/rust-rdkafka/issues/611

[slinkydeveloper]

Unfortunately I only have 3 workarounds in my mind:

• Set session.timeout.ms by default to 6000, which is the default minimum allowed: https://docs.confluent.io/platform/current/installation/configuration/broker-configs.html#group-min-session-timeout-ms. This is just a mitigation, it doesn't fix the issue completely.
• Stop using the auto offset commit feature, and manually sync offset commits. This might have a significant impact on the performance.
• Wrap Consumer in ManualDrop and not drop it only when shutting down Restate. Probably the unsafest of the proposed workarounds, as it is currently unclear to me what are the consequences, and still would not work when the user shutdowns a consumer task (e.g. because it removed a subscription) without shutting down Restate.

[slinkydeveloper]

A fourth alternative could be the following:

• We still asynchronously commit the offsets, but we do so on our own consumer task, by using commit alone https://docs.confluent.io/platform/current/clients/librdkafka/html/rdkafka_8h.html#ab96539928328f14c3c9177ea0c896c87

[slinkydeveloper]

While trying to go with the fourth alternative, I hit this other bug https://github.com/fede1024/rust-rdkafka/issues/597#issuecomment-1733828914. WIP: https://github.com/slinkydeveloper/restate/tree/shutdown_kafka